<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sushmitha Pulagam on Medium]]></title>
        <description><![CDATA[Stories by Sushmitha Pulagam on Medium]]></description>
        <link>https://medium.com/@sushmithapulagam?source=rss-311a5d2c608f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tehvSG6rz7ff0loM2HZPQw.jpeg</url>
            <title>Stories by Sushmitha Pulagam on Medium</title>
            <link>https://medium.com/@sushmithapulagam?source=rss-311a5d2c608f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:32:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sushmithapulagam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Feature Scaling — Effectively Choose Input Variables Based on Distributions]]></title>
            <link>https://medium.com/data-science/feature-scaling-effectively-choose-input-variables-based-on-distributions-3032207c921f?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3032207c921f</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[normal-distribution]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Sat, 01 Aug 2020 17:18:58 GMT</pubDate>
            <atom:updated>2020-08-01T17:32:13.246Z</atom:updated>
            <content:encoded><![CDATA[<h3>Feature Scaling — Effectively Choose Input Variables Based on Distributions</h3><h4>Demonstrating how to wisely choose the numerical variables for scaling which results in boosting the accuracy of the model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O8ioOmFVJPwH1o3_7Ojyjg.jpeg" /><figcaption>Photo by Siora Photography on Unsplash</figcaption></figure><p><em>We often come across a situation dealing with a variety of numerical variables consisting of different ranges, units, and magnitudes while building an ML model. As a common practice, we will apply Standardization or Normalization techniques for all the features before building a model. However, it is crucial to study the distributions of the data before making a decision on which technique to apply for feature scaling.</em></p><p>In this article, we will go through the difference between Standardization and Normalization along with understanding the distributions of the data. In the end, we will see how to select the strategies based on Gaussian and Non-Gaussian distribution of the features to improve the performance of the Logistic Regression model.</p><h4>Standardization Vs Normalization</h4><p>Both these techniques are sometimes used interchangeably but they refer to different approaches.</p><p><strong><em>Standardization</em></strong>: This technique transforms the data to have a mean of zero and a standard deviation to 1.</p><p><strong><em>Normalization</em></strong>: This technique transforms the values in variables between 0 and 1.</p><p>We are using the Pima Indian Diabetes dataset and you can find the same [<a href="https://github.com/SushmithaPulagam/FeatureScaling-with-Distributions">here</a>]</p><pre>import pandas as pd<br>import numpy as np<br>data = pd.read_csv(“Pima Indian Diabetes.csv”)<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*H6yOVm_TCOksy8upqY1Nxg.png" /><figcaption>First few records of the dataset</figcaption></figure><p>From the above, we can see that the numerical variables are varying in different ranges and the Outcome is the target variable. We will perform both the scaling techniques and apply Logistic Regression.</p><p>👉 <strong>Applying Standardization to all features and modeling.</strong></p><p>From the sklearn library, we need to use StandardScaler to implement Standardization.</p><pre>from sklearn.preprocessing import StandardScaler<br>Y = data.Outcome<br>X = data.drop(&quot;Outcome&quot;, axis = 1)<br>columns = X.columns<br>scaler = StandardScaler()<br>X_std = scaler.fit_transform(X)<br>X_std = pd.DataFrame(X_std, columns = columns)<br>X_std.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*SVp5VZllku1L_K91p1EF6Q.png" /><figcaption>Transformation of Input features after applying Standardization</figcaption></figure><p>Let us do the train and test the split for the standardized features.</p><pre>from sklearn.model_selection import train_test_split<br>x_train, x_test, y_train, y_test = train_test_split(X_std, Y, test_size = 0.15, random_state = 45)</pre><p>Now we are going to apply Logistic Regression on the standardized dataset.</p><pre>#Building Logistic Regression model on the Standardized variables<br>from sklearn.linear_model import LogisticRegression<br>lr_std = LogisticRegression()<br>lr_std.fit(x_train, y_train)<br>y_pred = lr_std.predict(x_test)<br>print(&#39;Accuracy of logistic regression on test set with standardized features: {:.2f}&#39;.format(lr_std.score(x_test, y_test)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*HVgQlbPe1vBDzEnp-lUczQ.png" /><figcaption>Accuracy of the model with standardized features</figcaption></figure><p>From the above, we can see that the accuracy of the model with all the features applying Standardization technique is 72 percent.</p><p>👉 <strong>Applying Normalization to all features and modeling.</strong></p><p>From the sklearn library, we need to use MinMaxScaler to implement Normalization.</p><pre>from sklearn.preprocessing import MinMaxScaler<br>norm = MinMaxScaler()<br>X_norm = norm.fit_transform(X)<br>X_norm = pd.DataFrame(X_norm, columns = columns)<br>X_norm.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*xyyo1NywvjlqtCsYMfsn7Q.png" /><figcaption>Transformation of Input features after applying Normalization</figcaption></figure><p>Let us do the train and test the split for the normalized features.</p><pre># Train and Test split of Normalized features<br>from sklearn.model_selection import train_test_split<br>x1_train, x1_test, y1_train, y1_test = train_test_split(X_norm, Y, test_size = 0.15, random_state = 45)</pre><p>Applying Logistic Regression on the normalized dataset.</p><pre>#Building Logistic Regression model on the Normalized variables<br>from sklearn.linear_model import LogisticRegression<br>lr_norm = LogisticRegression()<br>lr_norm.fit(x1_train, y1_train)<br>y_pred = lr_norm.predict(x1_test)<br>print(‘Accuracy of logistic regression on test set with Normalized features: {:.2f}’.format(lr_norm.score(x1_test, y1_test)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*bJTfAnn8GZYZf2VatSO4vQ.png" /><figcaption>Accuracy of the model with normalized features</figcaption></figure><p>The accuracy of the model when all the features are normalized is 74 percent.</p><p>👉 <strong>Understanding the distribution of features</strong></p><p>Let us plot the histograms of the variables to study the distribution.</p><pre># Plotting the histograms of each variable<br>from matplotlib import pyplot<br>data.hist(alpha=0.5, figsize=(20, 10))<br>pyplot.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*iz3KdpLBpBxYGVFiFqEb2Q.png" /><figcaption>Histogram for each feature to understand the distribution</figcaption></figure><p><strong><em>Gaussian Distribution</em></strong> — BMI, BloodPressure, Glucose.</p><p><strong><em>Non-Gaussian Distribution</em></strong> — Age, DiabetesPedigreeFunction, Insulin, Pregnancies, SkinThickness</p><h4>👉Normalize Non-Gaussian features and Standardize Gaussian-like features</h4><p>Finally, we came to an experiment waiting to select the variables and apply both the strategies based on the distributions on the same dataset.</p><p>To apply this strategy, we are going to use Column Transformer and Pipeline concepts from sklearn as we need to do the mixed type of techniques by subsetting the columns.</p><p>As mentioned above, we are initiating different pipelines for Gaussian and Non-Gaussian features</p><pre>from sklearn.compose import ColumnTransformer<br>from sklearn.pipeline import Pipeline<br><strong>Standardize_Var = [&#39;BMI&#39;,&#39;BloodPressure&#39;, &#39;Glucose&#39;]</strong><br>Standardize_transformer = Pipeline(steps=[(&#39;standard&#39;, StandardScaler())])<br><strong>Normalize_Var = [&#39;Age&#39;,&#39;DiabetesPedigreeFunction&#39;,&#39;Insulin&#39;,&#39;Pregnancies&#39;,&#39;SkinThickness&#39;]</strong><br>Normalize_transformer = Pipeline(steps=[(&#39;norm&#39;, MinMaxScaler())])</pre><p>Now, let us build the Logistic Regression model on the data with selective features for Standardization and Normalization.</p><pre>x2_train, x2_test, y2_train, y2_test = train_test_split(X, Y, test_size=0.2)<br><strong>preprocessor = ColumnTransformer(transformers=<br> [(‘standard’, Standardize_transformer, Standardize_Var),<br> (‘norm’, Normalize_transformer, Normalize_Var)])</strong></pre><pre>clf = Pipeline(steps=[(‘preprocessor’, preprocessor),<br> (‘classifier’, LogisticRegression(solver=’lbfgs’))])<br>clf.fit(x2_train, y2_train)<br>print(‘Accuracy after standardizing Gaussian distributed features and normalizing Non-Gaussian features: {:.2f}’.format(clf.score(x2_test, y2_test)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*AmP4sDfItcub4xY0VpshgQ.png" /></figure><p>👉 <strong>Final key details</strong></p><p>Below is the accuracy details for the different models we have built so far.</p><p>Accuracy after standardizing all the features: <strong>0.72</strong></p><p>Accuracy after normalizing all the features: <strong>0.74</strong></p><p><strong><em>Accuracy after applying Standardization to Gaussian distribution features and Normalization to Non- Gaussian distribution features: 0.79</em></strong></p><h4>Summary</h4><p>We need to perform Feature Scaling when we are dealing with Gradient Descent Based algorithms (Linear and Logistic Regression, Neural Network) and Distance-based algorithms (KNN, K-means, SVM) as these are very sensitive to the range of the data points. This step is not mandatory when dealing with Tree-based algorithms.</p><p>The main focus of this article is to explain how the distribution of the data plays an important role in feature scaling and how to select the strategies based on Gaussian and Non-Gaussian distribution to improve the overall accuracy of the model.</p><p>You can get the complete code from my GitHub [<a href="https://github.com/SushmithaPulagam/FeatureScaling-with-Distributions">profile</a>]</p><p>Thank you for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3032207c921f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-scaling-effectively-choose-input-variables-based-on-distributions-3032207c921f">Feature Scaling — Effectively Choose Input Variables Based on Distributions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to choose between a Linear or Nonlinear Regression for your dataset]]></title>
            <link>https://medium.com/data-science/how-to-choose-between-a-linear-or-nonlinear-regression-for-your-dataset-e58a568e2a15?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e58a568e2a15</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Fri, 17 Jul 2020 14:17:03 GMT</pubDate>
            <atom:updated>2020-07-17T14:17:03.725Z</atom:updated>
            <content:encoded><![CDATA[<h4>General guidelines to choose between Linear and Nonlinear regression to get a better fit and accuracy for your model.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrwqXTXtVDbP1H_AcZ3wwQ.jpeg" /><figcaption>Photo by Justin Luebke on Unsplash</figcaption></figure><p><em>Any Data Scientist starts their journey with regression analysis because it forms the basics of all predictive modeling and one can easily connect with the math and stats behind the scenes which is essential for any Machine Learning model. It is also necessary to know the different varieties of regression techniques to implement on your dataset to attain good accuracy with minimum error rate </em>✌️</p><p>In this article, we will go through a few plots and summary to determine which regression suits for our data set. Also, this article assumes that you have a prior knowledge of regression concept and its assumptions.</p><h4>👉<strong>Linear Regression</strong></h4><p>Linear Regression is a very common type of model used for predictive analysis for continuous data. Below is the equation for linear regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*Eg6HM1Yg_JWdU_Q5q4yB5g.png" /><figcaption>Linear Regression Equation</figcaption></figure><p>Let us go through an example with the below sample dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*8HVg1qbmSBfOjIIGxOBuMA.png" /><figcaption>Dataset for linear regression</figcaption></figure><p>Based on the No of weeks (X variable) we need to predict the number of cars sold.</p><p>Below is the scatter plot for the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*GQ6MHlJ2budxkxBsX9u9kA.png" /><figcaption>Linear Trend line for the scatter plot</figcaption></figure><p>From the above original scatter plot, we can see that there is a curvature at the end and the linear trend line is not doing justice for this dataset. Let’s build a regression model and interpret the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*Ckq43MGDtXLSZRWBRGM7Ag.png" /><figcaption>Summary output for Linear regression output</figcaption></figure><p>From the above output, the R Square is 0.80 and the Standard Error is 45. We will try to examine the Residual plot as well.</p><h4>👉Residual Plot for Linear Model</h4><p>The difference between the observed value and the mean value of an observation is called a residual. These are very important for regression because they indicate the extent to which the model accounts for the variation in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*YOdkJCDyiOlzipXKJEyBNg.png" /><figcaption>Residuals Plot forming a curve violating linear regression assumption</figcaption></figure><p>From the above, we can see that the residuals are forming a curve pattern which is a violation of one of the major assumptions for a linear model. The residuals should form a randomness but not any type of pattern which leads to predict one error based on the other. Hence, we can determine that the linear model is not suitable for our dataset.</p><h4>👉Nonlinear Regression</h4><p>Polynomial regression is an extension of linear regression where it fits a curvilinear relationship between target and independent variables.</p><p>Polynomial regression adds extra independent variables that are the powers of the original variable. Here, we are taking the degree as 2 and hence building a Quadratic regression model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*xwkLnx3fCVewQjhvCfe3VQ.png" /><figcaption>Nonlinear regression equation</figcaption></figure><p>Below is the data set created for the Quadratic regression model. An extra independent variable has been added which is No of Weeks*2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*UrX7xAFKyxgUuvnsZb1dQg.png" /><figcaption>No of Weeks *2 is created for Quadratic Regression model</figcaption></figure><p>Let’s do a scatter plot and draw a polynomial trendline to check how best the line fits the curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*RHzV3l1SuPUaIM7rGXyRFw.png" /><figcaption>Polynomial trend line over the scatter plot</figcaption></figure><p>From the above, we can see that the Polynomial trend line is trying to fit the curvature better than a straight line.</p><p>Let us build a model for the Nonlinear dataset we have created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*-hqMrlxVEZdRDwMJ-jzMHA.png" /><figcaption>Summary output for the Non-linear regression model</figcaption></figure><p>From the above output, we can see that the overall R Square value has increased which is 0.90 with a minimized standard error.</p><h4>👉<strong>Residual Plot for Quadratic Model</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*_ihjYlsyIlBde5a679b5Cg.png" /><figcaption>Residual Plot for Quadratic model with random errors</figcaption></figure><p>From the above plot, we can see that there is a randomness in the residuals, explaining the variance and hence satisfying the assumption of the regression model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*jKBQOld58HD-YU7474ydHw.png" /><figcaption>Comparing the Linear and Nonlinear regression outputs</figcaption></figure><h3>Summary</h3><p>In this article, we learned how the non-linear regression model better suits for our dataset which is determined by the non-linear regression output and residual plot.</p><p>Before building any regression model it is very important to review the scatter plots and check the tighter fit of the observations around the regression lines. Also, residual plots play a vital role in decision making as well. However, one should keep in mind that adding more independent variables to non-linear regression can overfit the model. Therefore, utmost care is required while adding more variables to non-linear regression models.</p><p>Thank you for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e58a568e2a15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-choose-between-a-linear-or-nonlinear-regression-for-your-dataset-e58a568e2a15">How to choose between a Linear or Nonlinear Regression for your dataset</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series forecasting using Auto ARIMA in python]]></title>
            <link>https://medium.com/data-science/time-series-forecasting-using-auto-arima-in-python-bb83e49210cd?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb83e49210cd</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Fri, 26 Jun 2020 18:42:33 GMT</pubDate>
            <atom:updated>2020-06-27T01:57:00.174Z</atom:updated>
            <content:encoded><![CDATA[<h4>Demonstration on how to leverage Auto ARIMA functionality in python using ‘pmdarima’ package to forecast the future</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QKcLHePKOSLThuMkzE0keg.jpeg" /><figcaption>Photo by Isaac Smith on Unsplash</figcaption></figure><p><em>Time Series analysis is one of the core skills of any data scientist and anyone working in the analytics field often comes across the situation to forecast the future provided with the past and present data.</em></p><p><strong>What is a Time Series?</strong></p><p>It is a series of observations taken at specified times basically at equal intervals. It is used to predict future values based on past observed values. The components you might observe in the time-series analysis are Trend, Seasonal, Irregular, and Cyclicity.</p><p>In the case of such datasets where only one variable is observed at each time is called ‘Univariate Time Series’ and if two or more variables are observed at each time is called ‘Multivariate Time Series’.</p><p>In this article, we will focus on the univariate time series for forecasting the sales with Auto ARIMA functionality in python which is almost similar to Auto ARIMA in R.</p><p><strong>Why use Auto ARIMA?</strong></p><p>Usually, in the basic ARIMA model, we need to provide the p,d, and q values which are essential. We use statistical techniques to generate these values by performing the difference to eliminate the non-stationarity and plotting ACF and PACF graphs. In Auto ARIMA, the model itself will generate the optimal p, d, and q values which would be suitable for the data set to provide better forecasting.</p><p>Let’s get started with installation…</p><p>Please note that this package was first named as “Pyramid” and later changed to “pmdarima”. Make sure you are installing the “pmdarima” package.</p><p>👉<strong>Step1: Installing ‘pmdarima’ package</strong></p><p>You can install this package from your Jupyter cell with the below command.</p><p><strong>! pip install pmdarima</strong></p><p>To ensure the package was built correctly, use the below command to test.</p><p><strong>from pmdarima.arima import auto_arima</strong></p><p>Also note that this package depends on several other python packages and to know more about the setup, refer to this [<a href="https://alkaline-ml.com/pmdarima/setup.html">link</a>]</p><p>The dataset we are using is monthly wise Champagne sales ranging from 1964 to 1972. You can get the dataset [<a href="https://github.com/SushmithaPulagam/TimeSeries_Auto-ARIMA">here</a>]</p><p>👉<strong>Step2: Preparing the data</strong></p><p>After reading the dataset, make sure you do not have null values and change the data type of the ‘Month’ variable to a datetime object. Also, set the index of the dataframe to this variable using the set_index method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*2K__aW1LIYPmglWZiyUjTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/269/1*lyKXTppy5jZkULtqFPC1SQ.png" /><figcaption>First few rows of sales data</figcaption></figure><p>👉<strong>Step3: Understanding the pattern</strong></p><p>To understand the pattern of the data, we can simply plot using .plot() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*GplDXfQ6Vo07Z2IG53UXUQ.png" /><figcaption>Seasonal pattern in the sales data</figcaption></figure><p>From the above, we can observe that there is a seasonality associated with the data. Every year there is a spike in increasing sales which continued from 1964 to 1972.</p><p>👉<strong>Step4: Test for Stationarity</strong></p><p>Stationarity is an important concept in time-series and any time-series data should undergo a stationarity test before proceeding with a model.</p><p>We use the ‘Augmented Dickey-Fuller Test’ to check whether the data is stationary or not which is available in the ‘pmdarima’ package.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*omUYdHu1NVu9Z2QLpNnXJw.png" /></figure><p>From the above, we can conclude that the data is non-stationary. Hence, we would need to use the “Integrated (I)” concept, denoted by value ‘d’ in time series to make the data stationary while building the Auto ARIMA model.</p><p>👉<strong>Step5: Train and Test split</strong></p><p>Split into train and test datasets to build the model on the training dataset and forecast using the test dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*v7Z7xZMG0okIJO0f5_zENg.png" /><figcaption>Plotting Train and Test datasets</figcaption></figure><p>👉<strong>Step6: Building Auto ARIMA model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*64ZOjhR_jsZ9D-E51MKvBQ.png" /></figure><p><em>Auto-Regressive (p)</em> -&gt; Number of autoregressive terms.</p><p><em>Integrated (d)</em> -&gt; Number of nonseasonal differences needed for stationarity.</p><p><em>Moving Average (q)</em> -&gt; Number of lagged forecast errors in the prediction equation.</p><p>In the Auto ARIMA model, note that small p,d,q values represent non-seasonal components, and capital P, D, Q represent seasonal components. It works similarly like hyper tuning techniques to find the optimal value of p, d, and q with different combinations and the final values would be determined with the lower AIC, BIC parameters taking into consideration.</p><p>Here, we are trying with the p, d, q values ranging from 0 to 5 to get better optimal values from the model. There are many other parameters in this model and to know more about the functionality, visit this link [<a href="https://alkaline-ml.com/pmdarima/modules/generated/pmdarima.arima.auto_arima.html">here</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*qiXnklgrbqJaGppxGkr-pg.png" /><figcaption>Auto ARIMA model parameters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*FdO5orn8Y9YJTT_gOsP6yQ.png" /><figcaption>Tracing of Auto ARIMA model</figcaption></figure><p>Below is the summary of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*0xoYGyxFR-gsQDk5CVJ2yQ.png" /><figcaption>Summary of the ARIMA model</figcaption></figure><p>👉<strong>Step7: Forecasting on the test data</strong></p><p>Using the trained model which was built in the earlier step to forecast the sales on the test data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*dSnV_pUoK22wPv3KD7pghw.png" /><figcaption>Sales forecasted on the test data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*6ueg6whmUwBiP9NaN1_TWg.png" /><figcaption>Plotting Train vs Test vs Predicted sales</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*8uNF_5Z4NWrZogDraOHZ6w.png" /><figcaption>R2 score for test data set</figcaption></figure><p><strong>Summary</strong></p><p>In this article, we learned how to use the functionality of the Auto ARIMA model and this approach will come into handy if you would like to generate the p, d, and q values from the model itself. In the basic ARIMA or SARIMA model, you need to perform differencing and plot ACF and PACF graphs to determine these values which are time-consuming.</p><p>However, it is always advisable to go with statistical techniques and implement the basic ARIMA model to understand the intuitive behind the p,d, and q values if you are new to time series.</p><p>You can get the dataset and the complete code from my GitHub [<a href="https://github.com/SushmithaPulagam/TimeSeries_Auto-ARIMA">here</a>]</p><p>Thank you for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb83e49210cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/time-series-forecasting-using-auto-arima-in-python-bb83e49210cd">Time Series forecasting using Auto ARIMA in python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All you need to know about the PCA technique in machine learning]]></title>
            <link>https://medium.com/data-science/all-you-need-to-know-about-pca-technique-in-machine-learning-443b0c2be9a1?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/443b0c2be9a1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 17:06:53 GMT</pubDate>
            <atom:updated>2020-06-15T17:07:32.743Z</atom:updated>
            <content:encoded><![CDATA[<h3>All you need to know about PCA technique in Machine Learning</h3><h4>A detailed explanation of the most widely used dimensionality reduction technique</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mcr4o3ueq7E6vS37bwMzKw.jpeg" /><figcaption>Photo by Johnson Wang on Unsplash</figcaption></figure><p><em>Has anyone come across a situation dealing with a large number of variables while building a model and worried about poor accuracy and efficiency? If so, Principal Component Analysis (PCA) will come to your rescue </em>✌️</p><p><strong>What is Principal Component Analysis (PCA)?</strong></p><p>It is one of the most widely used dimension reduction techniques to transform the larger dataset into a smaller dataset by identifying the correlations and patterns with preserving most of the valuable information.</p><p><strong>The need for PCA?</strong></p><p>It is used to overcome feature redundancy in the dataset. Also, it aims to capture valuable information explaining high variance which results in providing the best accuracy. It makes the data visualizations easy to handle. It decreases the complexity of the model and increases computational efficiency.</p><p>There is a lot of confusion behind the concept of selecting the number of principal components. In this article, we go through all the steps and understand the logic to determine the number of components that need to be selected for our dataset.</p><p>We are going to use the Pizza dataset with 8 variables. You can find the dataset [<a href="https://github.com/SushmithaPulagam/PCA">here</a>].</p><p>Below are the first few records of the Pizza dataset. “Brand” is the target variable and the remaining are independent variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*V1AWMcrbiHj0mQo1V-cgIw.png" /><figcaption>First few records of Pizza dataset</figcaption></figure><p>Removing the “Brand” column as we need to do PCA only for independent variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*rjcfp_Vgf5rru7ZX7GUAGA.png" /><figcaption>After removing the “Brand” variable</figcaption></figure><p>Let’s go through the steps</p><p>👉<strong>Step 1: Standardization of data</strong></p><p>Before proceeding with PCA, we need to perform the standardization of the data.</p><p><em>Performing standardization is a crucial step because the original variables may have different scales. We need to bring them to a similar range to get reasonable covariance analysis.</em></p><p>From the sklearn library, we can use the below code to standardize the data.</p><pre>from sklearn.preprocessing import StandardScaler<br>df_std = StandardScaler().fit_transform(df)<br>df_std</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*Kho3wAL7xQbcazVUCO-SmA.png" /><figcaption>Output after applying standardization</figcaption></figure><p>👉<strong>Step 2: Computing covariance matrix with standardized data</strong></p><p><em>The covariance matrix represents the correlation between two variables. This helps us to understand which two variables are heavily dependent on each other and to capture bias and redundancy in the dataset.</em></p><p>If the entry in the matrix is with negative sign means they are indirectly proportional to each other. If the sign is positive means they are directly proportional.</p><pre>df_cov_matrix = np.cov(df_std.T)<br>df_cov_matrix</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*VEfX8Dizj7HNc-iktvUuXQ.png" /><figcaption>Output of the covariance matrix</figcaption></figure><p>👉<strong>Step 3: Calculating Eigenvectors and Eigenvalues on the covariance matrix</strong></p><p><em>These two algebraic formulations are always computed as a pair which is also known as Eigendecomposition helps to reduce the dimension space by compressing the data. The core of the Principal Component Analysis is built on these values.</em></p><p>Each Eigenvector will have a corresponding Eigenvalue and the sum of all the Eigenvalues represents the overall variance in the entire dataset. It is very important to compute the Eigenvalues because it explains where the maximum variance lies in the dataset.</p><p>To know more about Eigenvectors and Eigenvalues, visit this [<a href="https://medium.com/fintechexplained/what-are-eigenvalues-and-eigenvectors-a-must-know-concept-for-machine-learning-80d0fd330e47#:~:text=Eigenvectors%20and%20eigenvalues%20revolve%20around,to%20represent%20a%20large%20matrix.">link</a>]</p><pre>eig_vals, eig_vecs = np.linalg.eig(df_cov_matrix)</pre><pre>print(‘Eigenvectors \n%s’ %eig_vecs)<br>print(‘\nEigenvalues \n%s’ %eig_vals)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*pPVjoqAGV43HiiMEodT9pA.png" /><figcaption>Output of Eigenvectors and Eigenvalues</figcaption></figure><p>👉<strong>Step 4: Sorting the Eigenvalues list in decreasing order</strong></p><p>After completing Eigendecomposition, we need to order the Eigenvalues in a descending order where the first value is the most significant and thus forms our first principal component.</p><pre>eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]</pre><pre>print(‘Eigenvalues in descending order:’)<br>for i in eig_pairs:<br> print(i[0])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*NgEcauOVJwl6Pvvq1HJz7g.png" /><figcaption>Output of the sorting Eigenvalues</figcaption></figure><p>👉<strong>Step5: Selecting the number of Principal Components</strong></p><blockquote>The first principal component will capture most of the variance from the original variables and the second principal component captures the second highest variance and so on…</blockquote><pre>total = sum(eig_vals)<br>var_exp = [(i / tot)*100 for i in sorted(eig_vals, reverse=True)]<br>cum_var_exp = np.cumsum(var_exp)<br>print(“Variance captured by each component is \n”,var_exp)<br>print(“Cumulative variance captured as we travel with each component \n”,cum_var_exp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*bPBib2FIeMKBPJNVmaSXKw.png" /><figcaption>Output of the variance captured by each component</figcaption></figure><p>From the above, we can see that the first principal component (PC1) captures overall 60% of the variance followed by PC2 with a 32.7% variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*r1KU7plK6S-5329GbUK6pg.png" /></figure><p>From the cumulative variance, overall 92% is being captured by 2 components and 98% of the variance is being explained by the first 3 components. Hence, we can decide that the number of principal components for our dataset is 3.</p><p>We can also visualize the same through the below scree plot with a cumulative sum of the explained variance ratio.</p><pre>pca = PCA().fit(df_std)<br>plt.plot(np.cumsum(pca.explained_variance_ratio_))<br>plt.xlabel(‘No of components’)<br>plt.ylabel(‘Cumulative explained variance’)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*tvWxr11N0Th-S5Bq7w6mdQ.png" /><figcaption>Scree plot to determine the number of components</figcaption></figure><p>👉<strong>Step6: Creating Principal Components</strong></p><p>With all the steps mentioned above, we have determined the number of components needed for our dataset is 3 considering with maximum variance.</p><pre>from sklearn.decomposition import PCA<br>pca = PCA(n_components = 3)<br>pcs = pca.fit_transform(df_std)<br>df_new = pd.DataFrame(data=pcs, columns={‘PC1’,’PC2&#39;,’PC3&#39;})<br>df_new[‘target’] = df1[‘Brand’] <br>df_new.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*glGIezMYoMsSSRBKChRBdw.png" /><figcaption>Dataframe of a new dataset created with Principal components</figcaption></figure><p>PCA can be done only to the numerical variables. If you have categorical data, then you need to convert into numerical features before applying PCA. Also, note that these principal components are linear combinations of the original dataset. These components are not as readable and interpretable as original features.</p><p><strong>Summary</strong></p><p>In this article, we have gone through all the steps to understand how to select the number of principal components required for the dataset. One should keep in mind that the components need to be select with utmost care or there would be a chance of information loss.</p><p>This PCA technique can be applied and would be a great help if you are dealing with multicollinearity issues in your dataset. If you would like to know more about multicollinearity, please visit my previous blog [<a href="https://towardsdatascience.com/how-to-detect-and-deal-with-multicollinearity-9e02b18695f1?source=friends_link&amp;sk=693ee8780b09f4a19a0f2d6ac33533a7">here</a>]</p><p>You can get the dataset and the complete code from my Github [<a href="https://github.com/SushmithaPulagam/PCA">here</a>]</p><p>Thank you for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443b0c2be9a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/all-you-need-to-know-about-pca-technique-in-machine-learning-443b0c2be9a1">All you need to know about the PCA technique in machine learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to detect and deal with Multicollinearity]]></title>
            <link>https://medium.com/data-science/how-to-detect-and-deal-with-multicollinearity-9e02b18695f1?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e02b18695f1</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[multicollinearity]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 17:22:19 GMT</pubDate>
            <atom:updated>2020-06-06T17:22:19.783Z</atom:updated>
            <content:encoded><![CDATA[<h4>Detailed explanation on identifying and fixing multicollinearity for numerical variables</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AY_sT3Bw9UVmhctBqTtUoA.jpeg" /><figcaption>Photo by Jaxon Lott on Unsplash</figcaption></figure><p><em>Multicollinearity is one of the main assumptions that need to be ruled out to get a better estimation of any regression model </em>✌️</p><p>In this article, I’ll go through the impact of multicollinearity, how to identify, and when to fix this issue with a sample dataset.</p><p>👉 <strong>What is correlation?</strong></p><p>The correlation between two variables can be measured with a correlation coefficient which can range between -1 to 1. If the value is 0, the two variables are independent and there is no correlation. If the measure is extremely close to one of these values, it indicates a linear relationship and highly correlated with each other. This means a change in one variable is associated with a significant change in other variables.<br>The correlation can be calculated with the <a href="https://en.wikipedia.org/wiki/Pearson_correlation_coefficient">Pearson correlation coefficient</a> and <a href="https://en.wikipedia.org/wiki/Spearman&#39;s_rank_correlation_coefficient">Spearman rank-order coefficient</a>.</p><p>👉 <strong>What is Multicollinearity?</strong></p><p>Multicollinearity occurs when there is a high correlation between the independent variables in the regression analysis which impacts the overall interpretation of the results. It reduces the power of coefficients and weakens the statistical measure to trust the p-values to identify the significant independent variables. Hence, we would not be able to examine the individual explanation of the independent variables on the dependent variable.</p><p>👉 <strong>When to fix Multicollinearity?</strong></p><p>The good news is that it is not always mandatory to fix the multicollinearity. It all depends on the primary goal of the regression model.</p><p>The degree of multicollinearity greatly impacts the p-values and coefficients but not predictions and goodness-of-fit test. If your goal is to perform the predictions and not necessary to understand the significance of the independent variable, it is not a mandate to fix the multicollinearity issue.</p><p>👉 <strong>How to test Multicollinearity?</strong></p><ol><li><strong>Correlation matrix / Correlation plot</strong></li><li><strong>Variation Inflation Factor (VIF)</strong></li></ol><p><em>A correlation plot can be used to identify the correlation or bivariate relationship between two independent variables whereas VIF is used to identify the correlation of one independent variable with a group of other variables. Hence, it is preferred to use VIF for better understanding.</em></p><p>VIF = 1 → No correlation<br>VIF = 1 to 5 → Moderate correlation<br>VIF &gt;10 → High correlation</p><p>Let’s get started with code…..</p><p>Here is the link for the dataset [<a href="https://github.com/SushmithaPulagam/Fixing-Multicollinearity">House Sales</a>]</p><p>The features in the dataset are as below. Sale Price is the target variable and the remaining are independent features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*flGXkA9EEMwA_29-XsYTMA.png" /><figcaption>First few rows of the dataset</figcaption></figure><p><strong>Identifying Multicollinearity with numerical variables</strong></p><p>Now we will calculate the VIF scores for each independent variable.</p><pre>def vif_scores(df):<br>    VIF_Scores = pd.DataFrame()<br>    VIF_Scores[&quot;Independent Features&quot;] = df.columns<br>    VIF_Scores[&quot;VIF Scores&quot;] = [variance_inflation_factor(df.values,i) for i in range(df.shape[1])]<br>    return VIF_Scores</pre><pre>df1 = df.iloc[:,:-1]<br>vif_scores(df1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*hJGreWL1P5oV7hxouwOTLw.png" /><figcaption>VIF for each independent feature</figcaption></figure><p>The VIF scores are higher than 10 for most of the variables. The individual coefficients and the p-values will be greatly impacted if we build a regression model with this dataset. We will proceed on how to fix this issue.</p><p><strong>Fixing Multicollinearity — Dropping variables</strong></p><p>We will consider dropping the features <em>Interior(Sq Ft) </em>and <em># of Rooms</em> which are having high VIF values because the same information is being captured by other variables. Also, it helps to reduce the redundancy in the dataset.</p><p>Let us compare the VIF values before and after dropping the VIF values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*HmypHAiQ4n7jYI2TjumMTw.png" /><figcaption>The highlighted features are removed and calculated the VIF scores</figcaption></figure><p>From the above, we can notice that the VIF scores have reduced for other variables also after dropping the high-value (<em>Interior(Sq Ft) </em>and <em># of Rooms) </em>VIF features.</p><p><strong>Fixing Multicollinearity — Combining variables</strong></p><p>Next, we can observe that #of Bed and #of Bath can be combined to a single variable and it helps us to capture more information from both variables.</p><pre>df5 = df4.copy()<br>df5[&#39;Total Rooms&#39;] = df4.apply(lambda x: x[&#39;# of Bed&#39;] + x[&#39;# of Bath&#39;],axis=1)<br>X = df5.drop([&#39;# of Bed&#39;,&#39;# of Bath&#39;],axis=1)<br>vif_scores(X)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*GmjAaab6tgNfPJBFtThXXg.png" /><figcaption>#of Bed and #of Bath features combined to a single feature Total Rooms</figcaption></figure><p>From the above, we can notice that all three variables (Condo Fee, Tax, and Total Rooms) came up with satisfying VIF values and we can proceed further to build a regression model.</p><p><strong>Summary</strong></p><p>In this article, we learned how to identify and the ways to fix the multicollinearity issue with numerical values in regression analysis. This is an iterative process and we also need domain knowledge to decide on which variables to take an appropriate action. There are other techniques such as PCA and Regularization methods also to address this issue.</p><p>Please visit my GitHub link for complete [<a href="https://github.com/SushmithaPulagam/Fixing-Multicollinearity">code</a>]</p><p>Thank You for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e02b18695f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-detect-and-deal-with-multicollinearity-9e02b18695f1">How to detect and deal with Multicollinearity</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Simplified approach using PyCaret for Anomaly Detection]]></title>
            <link>https://medium.com/data-science/a-simplified-approach-using-pycaret-for-anomaly-detection-7d33aca3f066?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d33aca3f066</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[pycaret]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Sun, 24 May 2020 17:28:42 GMT</pubDate>
            <atom:updated>2020-05-24T17:28:42.400Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explaining outlier detection with PyCaret package in python.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEt-YxTpvWqymh989hpKUA.jpeg" /><figcaption>Photo by Rupert Britton on Unsplash</figcaption></figure><p><strong>What is an Anomaly?</strong></p><p>As per the google definition “<strong><em>Something that deviates from what is standard, normal, or expected</em></strong>”. The same citation applies in the Data Science field and we come across such anomalies that need to be detected, treated, or eliminated to provide better business solutions or to build best predictive machine learning models. These anomalies are also known as “Outliers” in terms of a dataset and also referred to novelties, noise, deviations, and exceptions.</p><p><strong>Use cases of Anomaly detections</strong></p><p>There are a variety of domains that implement this technique such as fault detection, intrusion, fraud detection, health care monitoring, detecting ecosystem disturbances, and many more.</p><p>In this article, we are going to learn how to detect outliers with a few steps for a given data set with PyCaret library available in python.</p><p>👉<strong>Installing PyCaret library</strong></p><p>It is recommended to create a separate Conda environment before installing the PyCaret library. You might get some errors related to dependencies if you try to install it in your base environment. It was easily installed when I tried installing in a new separate Conda environment. To know the complete details of the installation by creating a new Conda env, click [<a href="https://pycaret.org/install/">here</a>]</p><p>To demonstrate the outlier detection with PyCaret library, I created a dataset and introduced some extreme values intentionally.</p><p><em>Before proceeding further, have a look at the Anomaly Detection Module documentation and how it works for better understanding [</em><a href="https://pycaret.org/anomaly-detection/"><em>here</em></a><em>]</em></p><p>👉<strong>Creating a dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*vvwKT9BwRvsNI4sQYKKiuA.png" /><figcaption>Outliers highlighted in the dataset</figcaption></figure><p>👉<strong>Import module and initializing setup()</strong></p><p>setup() function will take a mandatory data frame parameter and starts processing after execution. It infers all the datatypes of the variables automatically as ask us to confirm or quit. From the below, the data type inferred as categorical instead of numerical. We need to include another parameter to change the data type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*tIeSENAGiQxFMY0xcNZryA.png" /><figcaption>Entered quit as the datatype is not categorical</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*2wfCtCZxYLmG4DY5-QTTiw.png" /><figcaption>Setup completed after adding numeric_features parameter</figcaption></figure><p>👉<strong>Creating a model</strong></p><p>create_model() function is used for creating a model and there are a lot of pre-defined algorithms available. I have chosen the “Isolation Forest” estimator which is abbreviated as ‘iforest’. It returns the trained model object. They are ‘knn’, ‘PCA’, ‘SVM’, and many more models available to explore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*xhaP3J8PkePINyMqrL5Tnw.png" /><figcaption>Trained model using iforest</figcaption></figure><p>👉<strong>Visualizing the outliers</strong></p><p>Thanks to PyCaret for such wonderful 3d plots. From the below plot, we can see that yellow dots represent the outliers and these are labeled as 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*qtkc8bSWe7fHA3u0rsF2UA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*OXN-XZz5KCJJJgRIDBSsRA.png" /><figcaption>3D visualization plot representing labels</figcaption></figure><p>👉<strong>Assigning the labels</strong></p><p>assign_model() function returns data frame with Outlier flag (1 = outlier, 0 = inlier) and decision score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*6w5s8X2rPQLtRPJr3FlK5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*vPTncWvEImfm-2vzM2A0bA.png" /><figcaption>Outliers are represented with Label value as 1</figcaption></figure><p>Now we can see how PyCaret detects the outliers with ease. There are other functions also available for prediction, to save and deploy the model within this module. Please go through the documentation of Anomaly Detection module in PyCaret [<a href="https://pycaret.org/anomaly-detection/">here</a>]</p><p><strong>Summary</strong></p><p>There are a lot of statistical techniques and traditional methods available to detect outliers in the data science field. One should aware of these techniques to get the rationale behind this concept and to understand business nuances when dealing with outliers.</p><p>This article is written to demonstrate the power of the PyCaret library for anomalies detection with a sample dataset for better understanding. You can implement this approach with real data sets and in your projects to save time and to capture beautiful plots.</p><p><em>I became a fan of PyCaret library and I would like you to experience the same </em>😊</p><p>You can find the code here from my GitHub [<a href="https://github.com/SushmithaPulagam/Anomaly-Detection_PyCaret">link</a>]</p><p><strong>References</strong></p><ol><li><a href="https://pycaret.org/anomaly-detection/">https://pycaret.org/anomaly-detection/</a></li><li><a href="https://pycaret.org/">https://pycaret.org/</a></li><li><a href="https://www.youtube.com/watch?v=q0dxYDq1A40">https://www.youtube.com/watch?v=q0dxYDq1A40</a></li></ol><p>Thank You for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d33aca3f066" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-simplified-approach-using-pycaret-for-anomaly-detection-7d33aca3f066">A Simplified approach using PyCaret for Anomaly Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Step by Step approach of deploying ML model on Cloud with Azure]]></title>
            <link>https://medium.com/data-science/step-by-step-approach-of-deploying-ml-model-on-cloud-with-azure-b7139cde7683?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7139cde7683</guid>
            <category><![CDATA[python-flask]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Sun, 17 May 2020 16:37:49 GMT</pubDate>
            <atom:updated>2020-05-17T16:37:49.780Z</atom:updated>
            <content:encoded><![CDATA[<h4>A detailed way of explaining how to deploy ML models with Flask app linking GitHub repo to Azure</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjNSc7DN3q8DiCtFwz9cGw.jpeg" /><figcaption>Photo by Waldemar Brandt on Unsplash</figcaption></figure><p><em>Yay!! I built my first ML model, what do I do next?? How and where do I deploy it?</em>🙄</p><p>Any absolute beginner might end up with the above questions after building their first Machine Learning model and this article will demystify how to deploy your model on the cloud and check your results in real-time.</p><p>Here, I am going to walk you through a feasible way of deploying a model on Azure and once you get the web app URL, you can share that with anyone and it runs on the server round the clock😎</p><p>With a wide range of tools available these days there are many different ways to deploy a model. Below are the tools we are going to use in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*1vbwmdhYF9iqDsiK1AkFPg.png" /></figure><p>Below are the steps we are going to perform.</p><p>👉<strong>Step 1: Creating an ML model</strong></p><p>Here we are focusing on the steps to deploy but not to train the model. Hence, I have taken a sample data and built a Linear Regression model in Jupyter Notebook. The aim is to predict monthly grocery expenses when income and household size being independent variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*q8mzj0uoGBZ37fjhAs9_bg.png" /></figure><p>After completing the Linear Regression model, save it as .pkl file in write binary mode as below. For the complete dataset (Expenses_Prediction.csv) and code(LR_Model.py), click [<a href="https://github.com/SushmithaPulagam/Deploying-ML-on-Cloud">here</a>].</p><pre>pickle.dump(lr_model, open(‘E:\\lr_model.pkl’,’wb’))</pre><p>👉<strong>Step 2: Creating HTML form</strong></p><p>To predict the expenses, we need to collect the data from new input values provided in the form and then use our Linear Regression model to predict the output and display the result in the form. Hence, we are creating a simple HTML form and it will display as below in the browser.</p><p>You can find the code [<a href="https://github.com/SushmithaPulagam/Deploying-ML-on-Cloud">here</a>] named as index.html</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*WkpOiXzzTTOSEqTKTKsQSw.png" /></figure><p>👉<strong>Step 3: Creating a Flask app</strong></p><p>Flask is a framework used to build web applications in Python. To host the HTML form, we need to create a Flask web app. To install Flask, use the command !<em>pip install Flask</em></p><p>Below is the snapshot to understand the flask app code which is named app.py. You can get the code [<a href="https://github.com/SushmithaPulagam/Deploying-ML-on-Cloud">here</a>]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*UMuc7bjeRo7pj761CP0w3A.png" /></figure><p>Place all the files in the same folder and you can test this web app in your local system. Open Anaconda prompt, go to the files folder and type <strong><em>python app.py</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*MOiUrAOkGfOVqKWB-QgT-Q.png" /></figure><p>Copy the URL (<em>http://127.0.0.1:5000/</em>) and paste it in your web browser, hit enter. Once you provide the input values, you can predict the expenses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*iRDY8rAkH0iZf5vGCPZqSQ.png" /></figure><p>If you are able to make it up to here, now it is time to deploy the web app on the cloud.</p><p>👉<strong>Step 4: Creating configuration files.</strong></p><p>We need to create the below two configuration files to continue with deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/160/1*onmF7mxinecklJVtj8GBBg.png" /></figure><p><strong><em>Procfile</em></strong><em>:</em> Mention the below single line of code in the Procfile and this file should not have any extension. This is the file that is going to execute first. It uses Gunicorn which is a pure-Python HTTP server for WSGI applications and it acts as a liaison in between your web application and the webserver. Make sure to save the file without extension.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*7AejSFcNDd4j9vUKQlpwvA.png" /></figure><p><strong><em>Requirements.txt: </em></strong>Mention all your libraries and their dependencies in this .txt file along with their versions as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*4WqCszCIPER5hUhpukiH_w.png" /></figure><p>👉<strong>Step 5: Commit files in GitHub Repo.</strong></p><p>Create an account in <a href="https://github.com/">GitHub</a> and upload all files in a repository. If you aren&#39;t created a repository before, check this out [<a href="https://help.github.com/en/enterprise/2.13/user/articles/creating-a-new-repository">here</a>]</p><p>Make sure to put your index.html in the templates folder. You can view the files from my GitHub [<a href="https://github.com/SushmithaPulagam/Deploying-ML-on-Cloud">here</a>]</p><p>👉<strong>Step 6: Link GitHub Repo to Azure and Deploy.</strong></p><p>Create your free account in Azure with the link [<a href="https://azure.microsoft.com/en-in/free/">here</a>].</p><p>Follow the below steps to link GitHub repo to Azure</p><p>Once you sign in, click on “<em>App Services</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hzs3SYscoZcf5a3oFcTdGA.png" /></figure><p>Click on “<em>Add</em>”. Fill the details in the Web App as below. Create a new “<em>Resource Group</em>”. Enter “<em>Name</em>” in the Instance Details. Click on “<em>Review+Create</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*LTObTF4qvqttHVNolCN4wg.png" /></figure><p>After clicking on “<em>Create</em>”, you will see the below message. Click on “<em>Go to resource</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*JqgzFmhrRt8JkE2ZlxedLg.png" /></figure><p>Click the “<em>Deployment Center</em>” in the left pane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRHT3J8rsbp4_JqWMv99Qg.png" /></figure><p>Click on “<em>GitHub</em>”. For the first time connecting to GitHub, Azure will ask for authorization. Once connected, click on continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pp7sx1lDUVI6yWu5k-ijeA.png" /></figure><p>Click on “<em>App Service build service</em>” and continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*eYofXJIGRnYzVnD4n0ixcQ.png" /></figure><p>In the Code section, the “<em>Organization</em>” will automatically update after connecting to GitHub. Select the required “<em>Repository</em>” and continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*h0bY3eljUYQESg7UzyZ27Q.png" /></figure><p>Click on Finish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*BAs3vgwgmAceGgIhjlQBag.png" /></figure><p>It will take a couple of minutes to complete and meanwhile you will see the status as Running</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*WeWwAfhunIaidmLYgSpIjg.png" /></figure><p>After completion, you will see the status as “Success”. Click on “<em>Overview</em>” in the left pane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*g7ZEs-i3uOkYRZTqWIi9kQ.png" /></figure><p>Copy the “<em>URL</em>” from the top right corner and paste it in your web browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AzBdRMHqD8UxHDpA-EFcGw.png" /></figure><p>You can see the web URL is created with the Azure domain. Provide the input values and click on the Predict button. You can see the predicted expenses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*65N8OMY1WZ54ch70LONE7w.png" /></figure><p><em>Again Yayy!! Your Machine Learning model is running on the cloud now.</em>👍</p><p><strong>Summary</strong></p><p>The intent of this article is to make you understand the deployment with ease who is trying to take the first step towards deploying a model in the cloud. Later, you can explore various methods such as creating pipelines for automatic deployments and training the model on the cloud, etc. As a data science professional, one should be aware of the end-to-end approach to running a model.</p><p>You can get the complete code and files here from my GitHub <a href="https://github.com/SushmithaPulagam/Deploying-ML-on-Cloud">profile</a></p><p>Thank you for reading and Happy Learning! 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7139cde7683" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/step-by-step-approach-of-deploying-ml-model-on-cloud-with-azure-b7139cde7683">Step by Step approach of deploying ML model on Cloud with Azure</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Guide to Market Basket Analysis]]></title>
            <link>https://medium.com/swlh/a-beginners-guide-to-market-basket-analysis-dbf431e2e87a?source=rss-311a5d2c608f------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbf431e2e87a</guid>
            <category><![CDATA[market-basket-analysis]]></category>
            <category><![CDATA[apriori-algorithm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sushmitha Pulagam]]></dc:creator>
            <pubDate>Tue, 05 May 2020 12:45:43 GMT</pubDate>
            <atom:updated>2020-05-06T08:51:12.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Af6LO0y_CAhPviYOvoLEiA.jpeg" /></figure><p><em>Have you ever wondered why the items are placed in such a way in your local grocery stores or supermarkets? Based on the items purchased in your cart, how the online stores are suggesting other products making your online shopping experience feasible? How online/offline stores increasing their sales by offering promotions/discounts and coupons?</em></p><p><em>Well, we can get all these insights through Market Basket Analysis with the Apriori algorithm which internally works with the logic of Association rule mining.</em></p><p><strong>What are these Association rules?</strong></p><p>Association rules are combinations of items that occur together frequently in transactions. In a nutshell, it allows retailers to trace relationships between the items that are purchased by the people. The rule can be typically formed as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/0*xOoIYi4vF8JtJ0rK.png" /></figure><p>The above rule illustrates that customers who purchased butter and jam together are most likely to buy bread also. One might think that the above rule is quite obvious and why to use these algorithms to trace out. In real life, the transactional data could be enormous and it becomes extremely challenging to get insights by regular monitoring. We need to run these algorithms multiple times until we discover stronger rules which are beneficial for the retailer.</p><p><strong>Types of Association rules</strong></p><p><strong><em>Useful rules</em></strong> -&gt; These are the ones we need to filter out which gives actionable information with high quality.</p><p><strong><em>Trivial rules</em></strong> -&gt; These will be known by anyone who is familiar with the business.</p><p><strong><em>Inexplicable rules</em></strong>-&gt; These do not have any relative explanation and do not propose any course of action.</p><p>There is no hard way that we can automatically identify or segregate the transactions into the above-mentioned rules. This comes along the way with domain knowledge that put into practice.</p><p>In this article, we will go through all the steps on how to get the association rules with the Apriori algorithm along with the R code and visualizations.</p><p>Let us assume you have 1000 receipts on your table and we will discuss how to tackle the below questions.</p><p>✦ What are the frequent items that were sold?</p><p>✦ What items customers are purchasing together?</p><p>✦ What items customers are purchasing along with a specific item (Ex: Yogurt)?</p><p>To get the dataset, please click <a href="https://github.com/SushmithaPulagam/Market-Basket-Analysis-with-R">here</a></p><p>Each line is a transaction and each cell represents an item.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*jX2lso-Gp78HYecc.png" /><figcaption>Snapshot of groceries data from excel</figcaption></figure><p>Install and load the below libraries.</p><pre>install.packages(“arules”)<br>install.packages(“arulesViz”)<br>library(arules)<br>library(arulesViz)</pre><p>Read the data as a transaction object.</p><pre>groceries_data&lt;-read.transactions(choose.files(),format = &#39;basket&#39;,sep=&#39;,&#39;)<br>summary(groceries_data)</pre><p>From the summary, we can see the below output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/0*wzurj-76UdTEkLK4.png" /><figcaption>Summary of the groceries data</figcaption></figure><p>Let’s deep dive into more details from the above summary output.</p><p>The total number of items sold = transactions * items * density. 9835*169*0.0260 = 43215.</p><p>We can see that whole milk has the highest number of transactions, followed by other vegetables. The element length distribution tells us how many transactions are there with one item, 2 items set, and so on. There are 2159 transactions for a single item and 1643 transactions for 2 items and so on. Since the mean is a little higher than the median, we observe that the distribution of the data is positively skewed.</p><p>Now we will plot the top 20 frequent items from the data set.</p><pre>library(RColorBrewer)<br>itemFrequencyPlot(groceries_data,topN=20,type=”absolute”,col=brewer.pal(9,’Set3&#39;),main=’Items Frequency Plot’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*DG2N8iXORmq1A_Sg.png" /><figcaption>Frequency plot for the top 20 sold items</figcaption></figure><p>Whole milk stands at the top with the highest number of sales followed by other vegetables, rolls/buns, and so on. This graph helps us in identifying the rules for market basket analysis.</p><p><strong>How does the Apriori algorithm work?</strong></p><p>There are three statistical measures one should aware of before implementing this model.</p><p>➤ <strong><em>Support </em></strong>-&gt; It measures the frequency of Item A or the combination of frequency of items A and B together in the total number of transactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*-sT4fatW2O6MXheG.png" /><figcaption>ItemSet 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*AwNjAHVKZjlJCyos.png" /><figcaption>ItemSet 2</figcaption></figure><p>It is obvious that ItemSet 1 will have more support when compared to ItemSet2 because the combination of purchasing Bread and Pen is rare.</p><p>➤ <strong><em>Confidence</em></strong>-&gt;This measure tells us how often the items A and B occur together given the number of times A occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*wTZElxH-1uK3KSf6.png" /><figcaption>The ratio of transactions having Bread also had Butter in cart</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*7TiTvi7uw5Lmhdeg.png" /><figcaption>The ratio of transactions having Butter also had Bread in the cart</figcaption></figure><p>Here we have to note that ☟</p><p><em>Confidence({Bread} -&gt; {Butter}) </em><strong><em>≠ </em></strong><em>Confidence({Butter} -&gt; {Bread})</em></p><p>Support and Confidence are the two hyper tuning parameters in the Apriori algorithm we need to fine-tune until we get the stronger rules.</p><p>➤ <strong>Lift </strong>-&gt; This measure controls for the support (frequency) of the subsequent item while calculating the conditional probability of occurrence for {Butter} given {Bread}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/0*w3T6mrS63ntEzk_X.png" /></figure><p><strong>Model implementation with default values for parameters</strong></p><pre>association.rules &lt;- apriori(groceries_data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*kiCy3RouQsqahskq.png" /><figcaption>Output for the algorithm with default values</figcaption></figure><p>By default, the Apriori algorithm will consider <strong><em>Support as 0.1 and Confidence as 0.8. </em></strong>From the above output, we can see that no rules were written. Now it is time to tweak the parameters to get some rules.</p><p><strong>Model Implementation with different values for parameters</strong></p><pre>association.rules &lt;- apriori(groceries_data, parameter = list(<strong>supp=0.001</strong>, <strong>conf=0.8</strong>,maxlen=10))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*7G3FUIlbWSIE9vuG.png" /><figcaption>The output of the algorithm with Support =0.001 and Confidence =0.8</figcaption></figure><pre>summary(association.rules)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*Z6sev0CZhno70VPc.png" /><figcaption>Summary for the 410 rules generated</figcaption></figure><p>From the summary of association rules, we can see that 29 rules are generated with 3 items(lhs+rhs) and the length of 4 items(lhs+rhs) has the most rules i.e., 229.</p><p>All the 410 rules (including redundant) have a <strong>minimum lift of 3.131</strong> <strong>with the maximum being 11.235</strong>. Theoretically, the rules with lift &gt;1 are considered as good rules. But we will inspect the top 15 rules where lift &gt; 1 for further analysis after removing redundant rules.</p><p><strong>Removing redundant rules</strong></p><pre>subset.rules&lt;-which(colSums(is.subset(association.rules,association.rules))&gt;1)<br>length(subset.rules)<br>subset.asso.rules&lt;-association.rules[-subset.rules]<br>summary(subset.asso.rules)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*fjbA7Nn-wteQpTZ9.png" /><figcaption>Summary for the 319 rules generated</figcaption></figure><p>The length of redundant rules is 91. These 91 rules can be removed from 410 rules. <em>Post removing the redundant rules, we are left with 319 rules.</em></p><p><strong>Inspecting the first 15 rules out of 319 rules</strong></p><pre>inspect(subset.asso.rules[1:15])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/0*gL4AfN7Nw39DfEyi.png" /><figcaption>Inspection of the first 15 rules</figcaption></figure><p>For the 10th rule, the confidence is 1 and we can say that 100% of the customers who bought rice and sugar also bought whole milk.</p><p>We can also generate rules for a particular item. For Ex. Below is the code for generating rules only for yogurt.</p><pre>yogurt.asso.rules&lt;- apriori(groceries_data, parameter = list(supp=0.001, conf=0.8,maxlen=10),appearance = list(default = “lhs”,rhs = “yogurt”))<br>inspect(head(yogurt.asso.rules))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*yRrbs5eK9k8OEfc3.png" /><figcaption>Rules generated for yogurt</figcaption></figure><p><strong>Visualization Rules</strong></p><p>For an effective understanding of the rules and values for Support, Confidence and Lift, let us create few plots. Here, we are considering confidence &gt;0.4 and sorted with the confidence parameter.</p><pre>inspect(top10subRules[1:10]subRules&lt;-subset.asso.rules[quality(subset.asso.rules)$confidence&gt;0.4]<br>top10subRules &lt;- head(subRules, n=10, by= “confidence”)<br>plot(top10subRules, method = “graph”, html = “htmlwidget”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*UFjogCo2y1DOWGJL.png" /><figcaption>Inspection of top 10 rules to compare with the below graph</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*aj7niPJfVDxTDhsl.png" /><figcaption>Plot for top 10 rules</figcaption></figure><p>In the above plot, the size of the nodes is based on the highest <strong>Support</strong> and the color signifies the highest <strong>Lift</strong>. The incoming lines represent the LHS and the outgoing lines represent the RHS.</p><p>Below is the Parallel coordinates plot.</p><pre>subRules1&lt;-head(subRules,n=10,by=”lift”)<br>plot(subRules1, method = “paracoord”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/0*mDNkGKTCy83glcVI.png" /><figcaption>Parallel coordinates plot</figcaption></figure><p>So if a customer has red blush wine and liquor in the shopping cart, there are 11.23(lift) chances that s/he will buy beer along too.</p><p><strong>Summary</strong></p><p>This article is written to get familiar with the approach on how to perform Market Basket Analysis with sample data. Among all the rules, some might be useless. Hence, utmost care needs to be taken while selecting the values for support and confidence to get the stronger rules.</p><p>The association rules mining is not only limited to the Marketing domain, but the same can also be used in other domains such as Healthcare, a certain combination of health risk conditions can result in other complicated diseases. Credit card transactions, to know the likelihood of purchasing the next item based on the customer’s previous purchases and so on…</p><p>You can get the R code from my Github profile <a href="https://github.com/SushmithaPulagam/Market-Basket-Analysis-with-R">here</a>.</p><p>Thank you for reading and Happy Learning ☺</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbf431e2e87a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-beginners-guide-to-market-basket-analysis-dbf431e2e87a">A Beginner’s Guide to Market Basket Analysis</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>