<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tara Mullin on Medium]]></title>
        <description><![CDATA[Stories by Tara Mullin on Medium]]></description>
        <link>https://medium.com/@tarammullin?source=rss-cc29bf05b59b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*94t6YWjqFMVTFYlS.jpg</url>
            <title>Stories by Tara Mullin on Medium</title>
            <link>https://medium.com/@tarammullin?source=rss-cc29bf05b59b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:38:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tarammullin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Visualizing DBSCAN Results with t-SNE & Plotly]]></title>
            <link>https://medium.com/@tarammullin/visualizing-dbscan-results-with-t-sne-plotly-e3742205c900?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3742205c900</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[interactive-visualization]]></category>
            <category><![CDATA[dbscan]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 15:06:18 GMT</pubDate>
            <atom:updated>2020-07-15T15:06:18.530Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I experimented with a clustering algorithm called DBSCAN (Density-Based Spatial Clustering of Applications with Noise). DBSCAN is a clustering algorithm used to identify clusters of varying shape and size within a data set (Ester et al. 1996). I wrote a previous post describing DBSCAN, <a href="https://medium.com/@tarammullin/dbscan-2788cfce9389">here</a>, and DBSCAN parameter estimation, <a href="https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd">here</a>. Below I’ll describe how I explored my DBSCAN model results using t-SNE, a technique for dimensionality reduction, and Plotly, a tool for interactive visualization.</p><h4>t-SNE</h4><p>t-Distributed Stochastic Neighbor Embedding (t-SNE) is an unsupervised, non-linear technique used to visualize and explore high-dimensional data. It was introduces by <a href="https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf">van der Maaten and Hinton in 2008</a>. t-SNE creates a 2-D visual representation of multi-dimensional data while preserving local similarities and spacial relationships between data points in their original, higher dimensionality. Essentially, it can help us understand how data is distributed and arranged in high-dimensional space. For more thorough explanations, see the original paper <a href="https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf">here</a> or a great Towards Data Science article <a href="https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1">here</a>.</p><h4>Plotly</h4><p>Plotly is an interactive, open-source graphing library for Python. <a href="https://plotly.com/">Plotly</a> describes itself as “the front-end for ML and data science models”. Below is a very simple use case for Plotly. I’ve barely scratched the surface of this powerful library.</p><h4>Example</h4><p>I recently built my own DBSCAN model to identify clusters within World Bank development indicator data. The dataset I used contains measures for 10 different development indicators for every country for every year from 1990 to 2015.</p><p>After running DBSCAN, I used t-SNE and Plotly to visualize and explore the results.</p><p>Once I ran my DBSCAN model and appended the cluster labels (“labels”) to my original dataset (df), I then ran t-SNE using the same data array (data_array) I fed into the DBSCAN model. Scikit Learn has the t-SNE algorithm, documentation <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html">here</a>.</p><pre>from sklearn.manifold import TSNE</pre><pre>X_embedded = TSNE(n_components=2).fit_transform(data_array)</pre><p>Then, I appended the x and y components identified by t-SNE to my original dataset (df).</p><pre>df[&quot;x_component&quot;]=X_embedded[:,0]<br>df[&quot;y_component&quot;]=X_embedded[:,1]</pre><p>To build the interactive Plotly visualization I needed the following:</p><ul><li>X axis of plot = x component identified by t-SNE (“x_component”)</li><li>Y axis of plot = y component identified by t-SNE (“y_component”)</li><li>Color of each data point = cluster label identified by DBSCAN (“labels”)</li><li>Hover label = data point label (whatever you’d like to show up when you hover over a data point in the plot. In my case, this is country &amp; year)</li></ul><pre>import plotly.express as px<br>    <br>fig = px.scatter(df, x=&quot;x_component&quot;, y=&quot;y_component&quot;, hover_name=&quot;Country Year&quot;, color = &quot;labels&quot;, size_max=60)</pre><pre>fig.update_layout(<br>     height=800)</pre><pre>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3LmR0uzPFvjbAc03dCBzQ.png" /></figure><p>Now, when I hover over each data point, Plotly shows me which country in which year that point represents, as well as which cluster if falls into.</p><p><strong>Resources</strong> that helped me implement t-SNE and Plotly and write this post:</p><ul><li><a href="https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf">https://lvdmaaten.github.io/publications/papers/JMLR_2008.pdf</a></li><li><a href="https://lvdmaaten.github.io/tsne/">https://lvdmaaten.github.io/tsne/</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html">https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html</a></li><li><a href="https://distill.pub/2016/misread-tsne/">https://distill.pub/2016/misread-tsne/</a></li><li><a href="https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1">https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1</a></li><li><a href="https://plotly.com/python/getting-started/?utm_source=mailchimp-jan-2015&amp;utm_medium=email&amp;utm_campaign=generalemail-jan2015&amp;utm_term=bubble-chart">https://plotly.com/python/getting-started/?utm_source=mailchimp-jan-2015&amp;utm_medium=email&amp;utm_campaign=generalemail-jan2015&amp;utm_term=bubble-chart</a></li><li><a href="https://plotly.com/">https://plotly.com/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3742205c900" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DBSCAN — Overview, Example, & Evaluation]]></title>
            <link>https://medium.com/@tarammullin/dbscan-2788cfce9389?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2788cfce9389</guid>
            <category><![CDATA[python-data-science]]></category>
            <category><![CDATA[dbscan]]></category>
            <category><![CDATA[clustering-algorithm]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[silhouette-score]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 03:02:19 GMT</pubDate>
            <atom:updated>2020-07-15T15:26:25.897Z</atom:updated>
            <content:encoded><![CDATA[<h3>DBSCAN — Overview, Example, &amp; Evaluation</h3><h3>DBSCAN Overview</h3><p>Clustering is an unsupervised learning technique used to group data based on similar characteristics when no pre-specified group labels exist. This technique is used for statistical data analysis across many fields.</p><p>There are various algorithms that can be used to perform clustering. Recently, I experimented with a clustering algorithm called DBSCAN (Density-Based Spatial Clustering of Applications with Noise). DBSCAN is a density-based clustering algorithm used to identify clusters of varying shape and size with in a data set (Ester et al. 1996).</p><h4>Advantages of DBSCAN over other clustering algorithms:</h4><ol><li>DBSCAN does not require a pre-determined set number of clusters</li><li>DBSCAN identifies outliers as noise, instead of classifying them into a cluster.</li><li>DBSCAN is more flexible when it comes to the size and shape of clusters than other partitioning methods, such as K-means. It is able to identify clusters that differ in size and shape from one another, which makes it more useful for messy, real life data.</li></ol><h4>DBSCAN Parameters</h4><p>DBSCAN has two parameters. The first is ε, epsilon (“esp”), which defines the maximum distance allowed between two points within the same cluster. The second is minimum samples (“MinPts”), which defines the minimum number of data points required to form a distinct cluster. So, MinPts is the minimum number of neighbors contained within a cluster with radius/max length of esp. You can find a post I wrote about DBSCAN parameter estimation <a href="https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd">here</a>.</p><h3>Example model</h3><p>I recently built my own DBSCAN model. I chose DBSCAN primarily because you don’t need to specify the number of clusters. I didn’t know how many groups existed within the data set and identifying this number was one of the main points of my project. I used Scikit Learn to run the DBSCAN algorithm. See sklearn.cluster.DBSCAN documentation <a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html">here</a>.</p><pre>from sklearn import cluster</pre><p>I had previously estimated the DBSCAN parameters (more detail <a href="https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd">here</a>) MinPts = 20 and ε = 225. The data we put into DBSCAN should be “array-like” or “sparse matrix” in the shape of <em>(n_samples, n_features).</em></p><pre>eps = 225<br>min_samples = 20</pre><pre>dbscan = cluster.DBSCAN(eps=eps, min_samples=min_samples)<br>clustering_labels = dbscan.fit_predict(data_array)</pre><p>Append the results (clustering_labels) to the original dataframe — the dataframe the data array was derived from.</p><pre>df[&#39;labels&#39;] = clustering_labels</pre><h4>Evaluate Model Performance — Mean Silhouette Coefficient</h4><p>If the true cluster labels are unknown, as was the case with my data set, the model itself must be used to evaluate performance. An example of this type of evaluation is the Silhouette Coefficient.</p><p>The Silhouette Coefficient is bounded between 1 and -1. The best value is 1, the worst is -1. A higher score indicates that the model has better defined, more dense clusters. Values close to 0 indicate overlapping clusters, while negative values usually indicate that data points have been assigned to the wrong clusters. A paper about silhouettes can be found <a href="https://www.sciencedirect.com/science/article/pii/0377042787901257?via%3Dihub">here</a>.</p><p>Two scores are used to calculate the silhouette coefficient:</p><ul><li><strong>a:</strong> The average distance between one data point and all other points in the same cluster</li><li><strong>b:</strong> The average distance between one data point and all other points in the next nearest cluster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*d668xb6bBpGCLxGp3IObMA.png" /><figcaption>Formula to calculate the Silhouette Coefficient</figcaption></figure><p>I used SciKit Learn to calculate the Silhouette Coefficient. Documentation <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html#sklearn.metrics.silhouette_score">here</a>.</p><pre>from sklearn import metrics<br>metrics.silhouette_score(df, df[&#39;labels&#39;])</pre><p>My model received a Silhouette Coefficient score of 0.461. This is a decent score, indicating that my model doesn’t have overlapping clusters or mislabeled data points.</p><p><strong>Resources</strong> I learned a lot from while building my own DBSCAN model and writing this blog post:</p><ul><li><a href="https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf">https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf</a></li><li><a href="https://www.datanovia.com/en/lessons/dbscan-density-based-clustering-essentials/">https://www.datanovia.com/en/lessons/dbscan-density-based-clustering-essentials/</a></li><li><a href="https://towardsdatascience.com/how-dbscan-works-and-why-should-i-use-it-443b4a191c80">https://towardsdatascience.com/how-dbscan-works-and-why-should-i-use-it-443b4a191c80</a></li><li>Peter J. Rousseeuw (1987). “Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis”. Computational and Applied Mathematics 20: 53–65. <a href="https://doi.org/10.1016/0377-0427(87)90125-7">doi:10.1016/0377–0427(87)90125–7</a>.<a href="http://www2.cs.uh.edu/~ceick/7363/Papers/dbscan.pdf">http://www2.cs.uh.edu/~ceick/7363/Papers/dbscan.pdf</a></li><li><a href="https://scikit-learn.org/stable/modules/clustering.html#silhouette-coefficient">https://scikit-learn.org/stable/modules/clustering.html#silhouette-coefficient</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2788cfce9389" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DBSCAN Parameter Estimation Using Python]]></title>
            <link>https://medium.com/@tarammullin/dbscan-parameter-estimation-ff8330e3a3bd?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff8330e3a3bd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[k-nearest-neighbours]]></category>
            <category><![CDATA[clustering-algorithm]]></category>
            <category><![CDATA[nearest-neighbors]]></category>
            <category><![CDATA[dbscan]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 02:57:53 GMT</pubDate>
            <atom:updated>2020-07-15T15:27:06.518Z</atom:updated>
            <content:encoded><![CDATA[<p>DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is an unsupervised machine learning technique used to identify clusters of varying shape in a data set (Ester et al. 1996). Another post I wrote goes into what DBSCAN is and when to use it. You can find it <a href="https://medium.com/@tarammullin/dbscan-2788cfce9389">here</a>. This post will focus on estimating DBSCAN’s two parameters:</p><ol><li>Minimum samples (“MinPts”): the fewest number of points required to form a cluster</li><li>ε (epsilon or “eps”): the maximum distance two points can be from one another while still belonging to the same cluster</li></ol><h4>Minimum Samples (“MinPts”)</h4><p>There is no automatic way to determine the MinPts value for DBSCAN. Ultimately, the MinPts value should be set using domain knowledge and familiarity with the data set. From some research I’ve done, here are a few rules of thumb for selecting the MinPts value:</p><ul><li>The larger the data set, the larger the value of MinPts should be</li><li>If the data set is noisier, choose a larger value of MinPts</li><li>Generally, MinPts should be greater than or equal to the dimensionality of the data set</li><li>For 2-dimensional data, use DBSCAN’s default value of MinPts = 4 (Ester et al., 1996).</li><li>If your data has more than 2 dimensions, choose MinPts = 2*dim, where dim= the dimensions of your data set (Sander et al., 1998).</li></ul><h4>Epsilon (ε)</h4><p>After you select your MinPts value, you can move on to determining ε. One technique to automatically determine the optimal ε value is described in this <a href="https://iopscience.iop.org/article/10.1088/1755-1315/31/1/012012/pdf">paper</a>. This technique calculates the average distance between each point and its k nearest neighbors, where k = the MinPts value you selected. The average k-distances are then plotted in ascending order on a k-distance graph. You’ll find the optimal value for ε at the point of maximum curvature (i.e. where the graph has the greatest slope).</p><h4>Example</h4><p>Recently, I worked with a data set with 10 dimensions. Following the rules above, my minimum samples value should be at least 10. Ultimately, I decided to go with 20 (2*dim), as suggested in papers 1 and 2 listed under resources at the end of this post.</p><p>After I selected my MinPts value, I used NearestNeighbors from Scikit-learn, documentation <a href="https://ogrisel.github.io/scikit-learn.org/sklearn-tutorial/modules/generated/sklearn.neighbors.NearestNeighbors.html">here</a>, to calculate the average distance between each point and its n_neighbors. The one parameter you need to define is n_neighbors, which in this case is the value you choose for MinPts.</p><p>Step 1: Import Libraries</p><pre>from sklearn.neighbors import NearestNeighbors<br>from matplotlib import pyplot as plt</pre><p>Step 2: Calculate the average distance between each point in the data set and its 20 nearest neighbors (my selected MinPts value).</p><pre>neighbors = NearestNeighbors(n_neighbors=20)<br>neighbors_fit = neighbors.fit(dataset)<br>distances, indices = neighbors_fit.kneighbors(dataset)</pre><p>Step 3: Sort distance values by ascending value and plot</p><pre>distances = np.sort(distances, axis=0)<br>distances = distances[:,1]<br>plt.plot(distances)</pre><p>For my data set, the sorted distances produces a k-distance elbow plot that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*uQN5AxuaoPxgfxxwOlNkig.png" /><figcaption>Points sorted by distance to the 20th nearest neighbor</figcaption></figure><p>The ideal value for ε will be equal to the distance value at the “crook of the elbow”, or the point of maximum curvature. This point represents the optimization point where diminishing returns are no longer worth the additional cost. This concept of diminishing returns applies here because while increasing the number of clusters will always improve the fit of the model, it also increases the risk that overfitting will occur.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*r6H_Dq5W3AplWjpccIyEEQ.png" /><figcaption>Zoom in on “elbow” optimization point, points sorted by distance to the 20th nearest neighbor</figcaption></figure><p>Zooming in on my k-distance plot, it looks like the optimal value for ε is around 225. I ended up looping through combinations of MinPts and ε values slightly above and below the values estimated here to find the model of best fit.</p><p><strong>Resources</strong> that helped me estimate the parameters for my DBSCAN model and write this post:</p><ol><li><a href="https://link.springer.com/article/10.1023%2FA%3A1009745219419">https://link.springer.com/article/10.1023%2FA%3A1009745219419</a></li><li><a href="http://www.ccs.neu.edu/home/vip/teach/DMcourse/2_cluster_EM_mixt/notes_slides/revisitofrevisitDBSCAN.pdf">http://www.ccs.neu.edu/home/vip/teach/DMcourse/2_cluster_EM_mixt/notes_slides/revisitofrevisitDBSCAN.pdf</a></li><li><a href="https://www.datanovia.com/en/lessons/dbscan-density-based-clustering-essentials/">https://www.datanovia.com/en/lessons/dbscan-density-based-clustering-essentials/</a></li><li><a href="https://towardsdatascience.com/machine-learning-clustering-dbscan-determine-the-optimal-value-for-epsilon-eps-python-example-3100091cfbc">https://towardsdatascience.com/machine-learning-clustering-dbscan-determine-the-optimal-value-for-epsilon-eps-python-example-3100091cfbc</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff8330e3a3bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Confusion Matrices — Evaluating Machine Learning Classification Models]]></title>
            <link>https://medium.com/@tarammullin/machine-learning-classification-models-ff4964a11500?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff4964a11500</guid>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Fri, 24 Jan 2020 15:15:36 GMT</pubDate>
            <atom:updated>2020-07-10T00:51:45.005Z</atom:updated>
            <content:encoded><![CDATA[<h3>Confusion Matrices — Evaluating Machine Learning Classification Models</h3><p>You’ve built a classification model — great! But how accurately does the model classify? Confusion matrices allow us to visualize and evaluate how well the model classified a set of test data for which the true values are known.</p><p>It is simplest to use model that was trained for a binary classification task, meaning that each item is 1=True or 0=false.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*85ef0djUGP0IxfSvurpfEQ.png" /></figure><p><strong><em>True Positives (TP)</em></strong>:</p><ul><li>The model predicted true (1) and the real value is true (1).</li><li>Using an example from epidemiology, a diagnostic test indicated that a person has the disease (1), and they do actually have the disease (1).</li></ul><p><strong><em>True Negatives (TN)</em></strong>:</p><ul><li>The model predicted false (0) and the real value is false (0).</li><li>A diagnostic test indicated that a person is healthy (0), and they are actually healthy (0).</li></ul><p><strong><em>False Positives (FP)</em></strong>:</p><ul><li>The model predicted true (1), but the real value is false (0) — this is <strong>type I error.</strong></li><li>A diagnostic test indicated that a person has the disease (1), but they are actually healthy (0).</li></ul><p><strong><em>False Negatives (FN)</em></strong>:</p><ul><li>The model predicted false (0), but the real value was true (1)</li><li>A diagnostic test indicated that a person is healthy (0), but they actually have the disease (1).</li></ul><p>Confusion matrices become even more useful when they are used to calculate <strong>Evaluation Metrics</strong> such as <em>accuracy</em>, <em>precision</em>, and <em>recall.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*cRHf0ywVYXahesijPGP9sA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff4964a11500" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DC Real Estate Time Series Analysis]]></title>
            <link>https://medium.com/@tarammullin/dc-real-estate-time-series-analysis-df0b96e8ba2f?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/df0b96e8ba2f</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 23:41:25 GMT</pubDate>
            <atom:updated>2019-11-08T23:41:25.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*PluKpuCcwP9IxgeXfDG9Wg.png" /></figure><p>In November 2018, Amazon announced it’s HQ2 plans for Northern Virginia, which will employ 25,000 workers. Following the announcement, the <a href="https://www.washingtonian.com/2018/12/18/hq2-caused-a-700-increase-in-home-searches-around-dc-trulia/">Washingtonian</a> reported a 743 percent spike in online home searches in the area. The arrival of Amazon means that the region needs to add 267,000 housing units by 2025, according to an analysis by the <a href="https://www.washingtonpost.com/local/amazon-in-advanced-talks-about-putting-hq2-in-northern-virginia-those-close-to-process-say/2018/11/02/9be831d6-d7c0-11e8-aeb7-ddcad4a0a54e_story.html">Urban Institute</a>. However, the region is only on pace to to add about 170,000 units by then.</p><p>I’ve spent the past few weeks studying time series analysis. Given the DC area’s potential future housing shortage, I figured it could make an interesting case study to examine past prices, and attempt to build a model to predict future prices to determine <strong><em>which zip codes in DC are best to invest in</em></strong>. I used Zillow data on houses sold in DC between 1996 to 2018. You can find the dataset <a href="https://www.zillow.com/research/data/">here</a>.</p><h4>Data Preprocessing</h4><pre>import pandas as pd<br>zip_data = pd.read_csv(“zillow_data.csv”)<br>zip_data.head(15)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1d2sM-vrymDVSk2huRVxA.png" /></figure><pre>dc_zips = zip_data.loc[zip_data[&#39;State&#39;] == &#39;DC&#39;]<br>dc_zips.head()</pre><p>The database contained information on 18 of DC’s zip codes <em>(note the variable ‘RegionName’ here is zip code)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gillAosjCiAl8_DMNtTTSg.png" /></figure><pre>dc_zips.RegionName.unique()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cVPFzpyxk_vaiunNM-iD8w.png" /><figcaption>DC zip codes included in Zillow Data Set</figcaption></figure><p>For time series analysis, I needed to reshape the data from wide to long format using a function that calls the pandas melt function. Documentation <a href="https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.melt.html">here</a>.</p><pre>def melt_data(df):<br>    melted = pd.melt(df, id_vars=[&#39;RegionID&#39;,&#39;RegionName&#39;, &#39;City&#39;, &#39;State&#39;, &#39;Metro&#39;, &#39;CountyName&#39;,&#39;SizeRank&#39;], var_name=&#39;time&#39;)<br>    melted[&#39;time&#39;] = pd.to_datetime(melted[&#39;time&#39;], infer_datetime_format=True)<br>    melted = melted.dropna(subset=[&#39;value&#39;])<br>    return melted.groupby(&#39;time&#39;).aggregate({&#39;value&#39;:&#39;mean&#39;})</pre><p>I then created a dictionary containing data for each of the 18 zip codes melted into long format.</p><pre>zip_dict = {}<br>for z in dc_zips.RegionName.unique():<br>    temp_df = dc_zips[dc_zips.RegionName == z]<br>    zip_dict[z] = melt_data(temp_df)</pre><pre>zip_dict.items()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vaRUiI-cEiTI43zw7khlHw.png" /><figcaption>Data converted to long format</figcaption></figure><h4>Exploratory Data Visualization &amp; Analysis</h4><p>I next plotted all of the DC zip codes together on one line graph with price on the y-axis and year on the x-axis to get an overview of how prices within each zip code have changed over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*W05PkRUSBM8wqWuKcgCo-w.png" /><figcaption>Year vs. Average Price per Zip Code</figcaption></figure><p>I was interested in not only raw price changes, but also percent change over time so that I could understand which zip codes have seen the most growth since 1996. For each zip code, I created scaled versions with a percent change formula as follows: % change = (2018 price — 1996 price)/1996 price. I then plotted the scaled zip codes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZA_u_6vXEwYwHSK6YihCw.png" /><figcaption>Year vs. % Return per Zip Code</figcaption></figure><p>In deciding which zip codes are“best” to invest in, I considered the following factors within each zip:</p><ol><li>How much have average prices increased since 1996?</li><li>How much have average prices increased in the past few years?</li><li>How volatile are prices relative to other zips? — paying close attention to how prices were impacted by the recession.</li><li>How affordable is the current price?</li></ol><p>I ultimately selected <strong>20001</strong>, <strong>20002</strong>, <strong>20010</strong>, <strong>20003</strong>, and <strong>20009</strong> as the most promising DC zip codes included in the Zillow dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuH83iHh4jcRzILN1by-_g.png" /><figcaption>Year vs. Average Price of the 5 Top Zip Codes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XitThO6DrWbgIGAC2F0Tew.png" /><figcaption>Year vs. % Return of the Top 5 Zip Codes</figcaption></figure><p>Through this process of exploratory data analysis, I found that zip code <strong>20002</strong> was the most promising for investment because it has gained the second most value since 1996, has generally been less volatile, and is currently the most affordable of the top 5 zip codes. The second best zip code appears to be <strong>20003</strong>, as it has seen impressive and steady growth over time.</p><h4>SARIMAX Time Series Modeling</h4><p>In order to understand how the zip codes are predicted to perform in the future, I built a SARIMAX time series model. SARIMAX stands for <strong>Seasonal Autoregressive Integrated Moving Average</strong>. It is an extension of ARIMA (autoregressive integrated moving average), which is widely used in time series forecasting, but SARIMAX supports modeling of data containing seasonal trends by adding hyperparameters that address seasonal components. The documentation for SARIMX can be found <a href="https://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAX.html">here</a>.</p><p>The first step in fitting the SARIMAX model is to find the optimal values of the trend parameters p, d, and q of the ARIMA model.</p><pre>ARIMA(p,d,q)(P,D,Q)s</pre><blockquote><strong>Trend Elements</strong></blockquote><blockquote><strong>p</strong>: Trend autoregression order.</blockquote><blockquote><strong>d</strong>: Trend difference order.</blockquote><blockquote><strong>q</strong>: Trend moving average order.</blockquote><blockquote>— <a href="https://machinelearningmastery.com/sarima-for-time-series-forecasting-in-python/">Jason Brownlee, PhD</a></blockquote><p>I first found all the possible combinations of the p, d, and q parameters when they are allowed to each take on any value 0–2 using itertools.product and assuming s = 12 to represent 12 months.</p><pre>import itertools</pre><pre># Define the p, d and q parameters to take any value between 0 and 2<br>p = d = q = range(0,2)</pre><pre># Generate all different combinations of p, q and q triplets<br>pdq = list(itertools.product(p, d, q))</pre><pre># Generate all different combinations of seasonal p, q and q triplets (use 12 for frequency)<br>pdqs = [(x[0], x[1], x[2], 12) for x in list(itertools.product(p, d, q))]</pre><p>For evaluating the different combinations of parameters, I used the <strong>AIC (Akaike Information Criterion) value</strong>, which is provided by ARIMA models fitted using the`statsmodels` library. The Akaike information criterion (AIC) compares the quality of a collection of statistical models to one another, thus, providing a means for model selection.</p><p>A model that fits the data well while using lots of features will be assigned a larger AIC value than a model that uses fewer features to achieve the same goodness-of-fit. Therefore, we are interested in finding the model that yields the lowest AIC value.</p><p>An overview of AIC can be viewed <a href="https://www.statisticshowto.datasciencecentral.com/akaikes-information-criterion/">here</a>.</p><p>In the interest of brevity I will only show the modeling for zip code 20002, however, I did complete the remaining steps for each of the top 5 zip codes.</p><pre>#run a grid with all possible pdq and seasonal pdq combinations calculated above to find the lowest AIC value</pre><pre>ans = []<br>for comb in pdq:<br>    for combs in pdqs:<br>         try:<br>             mod = sm.tsa.statespace.SARIMAX(melted20002,<br>                                             order=comb,<br>                                             seasonal_order=combs,<br>                                             enforce_stationarity=False,<br>                                             enforce_invertibility=False)</pre><pre>output = mod.fit()<br>             ans.append([comb, combs, output.aic])<br>             print(&#39;ARIMA {} x {}12 : AIC Calculated ={}&#39;.format(comb, combs, output.aic))<br>         except:<br>             continue</pre><pre># Find the parameters with minimal AIC value.<br>ans_df = pd.DataFrame(ans, columns=[&#39;pdq&#39;, &#39;pdqs&#39;, &#39;aic&#39;])<br>ans_df.loc[ans_df[&#39;aic&#39;].idxmin()]</pre><p>The combination of parameter values that yielded the lowest AIC value were the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSUD9csoFnJgxulachhOTw.png" /></figure><p>Next, I plugged these parameters into a new SARIMAX model only fitting data up to year 2016 so that I could test the model’s predictive ability by comparing predictions to historic values for 2016 to 2018.</p><pre># plug the optimal parameter values into a new SARIMAX model.<br># Fit the model and print results<br>ARIMA_MODEL20002_1 = sm.tsa.statespace.SARIMAX(melted20002[:&#39;2016&#39;],<br>                                 order=(1, 1, 1),<br>                                 seasonal_order=(1, 1, 1, 12),<br>                                 enforce_stationarity=False,<br>                                 enforce_invertibility=False)<br>output20002_1 = ARIMA_MODEL20002_1.fit()</pre><pre>print(output20002_1.summary().tables[1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cmy0ri1z3n4eeQDsa2tGJg.png" /><figcaption>SARIMAX Results Table</figcaption></figure><p><strong>Validating the Model</strong></p><pre># Get predictions starting from 2016-01-01 (where the training left off) and calculate confidence intervals.</pre><pre>pred20002_1 = output20002_1.get_prediction(start=pd.to_datetime(&#39;2016-01-01&#39;), dynamic=False, end=pd.to_datetime(&#39;2018-01-01&#39;))<br>pred_conf20002_1 = pred20002_1.conf_int()<br></pre><pre># Plot real vs predicted values along with confidence interval</pre><pre>rcParams[&#39;figure.figsize&#39;] = 15, 6</pre><pre>#Plot observed values<br>ax = melted20002[&#39;2014&#39;:].plot(label = &#39;value&#39;)</pre><pre>#Plot predicted values<br>pred20002_1.predicted_mean.plot(ax=ax, label=&#39;One-step ahead forecast&#39;, alpha=.9)</pre><pre>#Plot the range for confidence intervals<br>ax.fill_between(pred_conf20002_1.index,<br>                pred_conf20002_1.iloc[:, 0],<br>                pred_conf20002_1.iloc[:, 1], color=&#39;g&#39;, alpha=.20)</pre><pre>#Set axes labels<br>ax.set_xlabel(&#39;Date&#39;)<br>ax.set_ylabel(&#39;Price&#39;)<br>plt.legend()</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUmGHh3T5XnYD4xYeJLl3A.png" /><figcaption>Year vs. Price Comparing Observed Values to One-Step Ahead Forecasted Values</figcaption></figure><p>The forecasted values produced by the model align with the true values from 2016 to 2017, with a tight confidence interval (shown in green). However, the forecast diverges from the true values during 2017 and the confidence interval increases greatly. The forecast predicted a larger increase in price than was observed in reality.</p><p>I also checked the accuracy of the forecasts by calculating the Mean Squared Error and Root Mean Squared Error. For each predicted value, I computed its distance to the true value and square the result. The results need to be squared so that positive/negative differences do not cancel each other out when we compute the overall mean. Then lastly, I took the square root. The Root Mean Squared Error of the model indicates that the model can predict values within + or — $16,820 of the actual value.</p><p>These are not fantastic results, but for my first SARIMAX time series model, I’ll take it! The model still requires a lot of work to improve it’s predictive ability, but it’s a decent start.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df0b96e8ba2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is effect size and why is it important?]]></title>
            <link>https://medium.com/@tarammullin/in-my-academic-experiences-with-statistics-and-hypothesis-testing-in-the-fields-of-natural-9c992be1e417?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c992be1e417</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Thu, 29 Aug 2019 18:18:21 GMT</pubDate>
            <atom:updated>2020-07-10T00:55:21.229Z</atom:updated>
            <content:encoded><![CDATA[<p>In my academic experiences with statistics and hypothesis testing in the fields of natural sciences and econometrics, there was a lot of focus on the <em>P</em> value. Until last week, I thought the <em>P</em> value was the single most important statistic to report when conducting hypothesis tests. However, recently I learned about another statistic of equal importance, which I can already understand may be more useful in my career as a data scientist. This statistic is <strong>effect size</strong>. An article that helped me understand this can be found <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/">here</a>. I will summarize some of the main points from this article and demonstrate an example of calculating effect size using Python below.</p><p><strong>What is effect size and why is it important?</strong></p><p>Effect size is the magnitude of difference between two groups you are comparing. Statistical significance, often evaluated with the <em>P</em> value, the probability that the findings are due to chance. While the <em>P </em>value is important in determining whether an effect or a difference exists, the <em>P</em> value does not tell you about the magnitude of the effect — this is where effect size is useful.</p><p>Another reason it is important to calculate and report effect size in addition to the <em>P </em>value has to do with sample size. While <em>P</em> values are dependent on sample size — with a very large sample size, a statistical test will likely demonstrate significant difference — effect size is independent of sample size.</p><p><strong>How is Effect Size Determined?</strong></p><p>There are several effect size indices. One of the most common, the one I used for my hypothesis tests project, is Cohen’s <em>d</em>. Cohen’s <em>d</em> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*jm9KkxK8qBGX-3l2-qHV9Q.png" /></figure><p>m_1 and m_2 are the respective sample means and s is the pooled standard deviation. It is the difference of means divided by the standard deviation. “<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/">The denominator standardizes the difference by transforming the absolute difference into standard deviation units (Sullivan and Feinn, 2012).</a>”</p><p>Generally, Cohen’s <em>d</em> values are interpreted as follows:</p><ul><li>Small Effect = 0.2</li><li>Medium Effect = 0.5</li><li>Large Effect = 0.8</li><li>Very Large Effect = 1.3</li></ul><p><strong>Calculating Effect Size Using Python</strong></p><p>Step 1: Import Numpy Library</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*bB7R58qEBqxdWoLSjFaYcQ.png" /></figure><p>Step 2: Define a Function to Calculate Cohen’s <em>d</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*RSfx4PVkUXIklbE5M8QqNw.png" /></figure><p>Step 3: Pass Two Comparison Groups Through the Function and Interpret Result</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*frAGx3jssmRvceW8Da9x_g.png" /></figure><p>My project was comparing data from Microsoft’s sample SQL database, Northwind, <a href="https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/linq/downloading-sample-databases">here.</a> I compared the quantity of products sold in orders where a discount was applied with orders without a discount. While I found a statistically significant P value when I compared the two groups with a t-test, calculating Cohen’s <em>d</em> helped me understand the impact of discount on quantity sold. Since d = 0.287, we can see that discount only has a small affect on quantity sold.</p><p>I reused the Cohen’s <em>d</em> function I defined above several times throughout my projects, which allowed me to compare the effect of a discount on various groups within the dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c992be1e417" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Data visualization for Exploratory Data Analysis (EDA)]]></title>
            <link>https://medium.com/@tarammullin/python-data-visualization-for-exploratory-data-analysis-eda-fafcf6ecdadc?source=rss-cc29bf05b59b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fafcf6ecdadc</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tara Mullin]]></dc:creator>
            <pubDate>Mon, 01 Jul 2019 23:39:46 GMT</pubDate>
            <atom:updated>2019-07-01T23:42:39.967Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Data Visualization for Exploratory Data Analysis (EDA)</h3><p>A little over two months ago, I began the <a href="http://https//flatironschool.com/career-courses/data-science-bootcamp/online/">Flatiron School’s Part Time Data Science</a> program. Through this program, I am learning a mix of software engineering and statistical understanding to gain the technical skills, expertise, and tools necessary to become a data scientist.</p><p>I just completed my first project and wanted to take a moment to reflect on one of the most time consuming, yet informative (and fun!) parts of my project — data visualization for Exploratory Data Analysis (EDA) using Python.</p><h3><strong>Overview of Module 1 Final Project</strong></h3><p>For the project we were given a dataset about house sales in King County, Washington. Our assignment was to clean, explore, and model this dataset with multivariate linear regression in order to predict the sale price of houses as accurately as possible.</p><p>The dataset included 20 features to describe things like square footage, location, quality, etc. In this blog post I will explore the following features through python visualizations:</p><blockquote>- Lat: latitude coordinate<br>- Long: longitude coordinate<br>- Sqft_living: Square footage of living space<br>- Grade: overall grade given to the house based on the King County grading system<br>- Waterfront: house which has a view to a waterfront<br>- Yr_built: the year the house was built<br>- Bedrooms: number of bedrooms in the house<br>- Bathrooms: number of bathrooms in the house<br>- Sqft_lot: sqare footage of the lot<br>- Sqft_living15: the square footage of the living space of the 15 nearest neighbors<br>- Sqft_above: Square footage of the house apart form the basement</blockquote><h3><strong>Getting Started</strong></h3><p>After I did some initial data cleaning, I wanted to gain more of an understanding of the dataset so I could make some predictions about which features may be best to include in my multiple regression model. That’s when I started some EDA. While creating EDA visualizations, I had the following goals in mind:</p><blockquote><strong>1. Assess the need for additional Data Cleaning </strong>— Which features need more cleaning, or to be dropped entirely?</blockquote><blockquote><strong>2. Determine Potential Features for the Model</strong> — Which features seem promising to include in a model to predict house prices?</blockquote><blockquote><strong>3. Discover Potential Limitations of the Model</strong> — What will be the limitations of my model may be based on this dataset?</blockquote><blockquote><strong>4. Evaluate Multicollinearity Between Predictors</strong> — Which features are correlated with each other and therefore problematic to include in a multiple regression model?</blockquote><h3><strong>Python EDA Visualizations</strong></h3><h4><strong>Scatter plots</strong></h4><p>Scatter plots use dots to display two variables in two dimensions — one along the x-axis and one along the y-axis. In the following examples I added a third dimension by coloring the dots by a third variable. I used latitude and longitude data in scatter plots to examine if there were any discernible patterns geographically. For example, where are the larger houses?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxajVG1NsAIJ2e5T85RiSw.png" /></figure><p>Or, where are the houses built with higher quality materials?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjzWiMOehA8kP6cg9IwoXg.png" /></figure><p>As we may expect, there is likely a relationship between location and price. The above scatter plots indicate that larger homes and homes with higher grade ratings are located in Central and North East King County. Ultimately, I did include zipcode in my model to capture this geographic element.</p><p>A scatter plot of latitude and longitude also helped me decide to ultimately drop a feature entirely. Waterfront was a boolean variable to indicate whether the house had a view of a waterfront.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*q-KJoHb_KMr4pSrLISWWIQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*U3GJ1Dq0USGyf8UBmRXtlw.png" /><figcaption><a href="https://www.google.com/maps/place/King+County,+WA/@47.5942007,-122.4036629,10z/data=!4m5!3m4!1s0x54905c8c832d7837:0xe280ab6b8b64e03e!8m2!3d47.5480339!4d-121.9836029">https://www.google.com/maps/place/King+County,+WA/@47.5942007,-122.4036629,10z/data=!4m5!3m4!1s0x54905c8c832d7837:0xe280ab6b8b64e03e!8m2!3d47.5480339!4d-121.9836029</a></figcaption></figure><p>Comparing our scatter plot of latitude vs. longitude colored by waterfront to a map of King County I pulled from Google Maps, I saw that there are actually many dots directly on top of each other or next to each other where one is coded as waterfront and one is not. In addition to other red flags I’d already identified with the variable, this scatter plot provided the final justification for excluding waterfront from my model.</p><h4><strong>Box plots</strong></h4><p>Box plots display 5-point statistics to help us understand the spread and centrality of data all in one visualization. Here’s what we can see in a box plot:</p><blockquote>- Minimum: minimum number in the data set<br>- Maximum: maximum number in the data set<br>- Median: the “middle” when the data set is sorted in ascending order<br>- First quartile: when the data set is sorted in ascending order, 25% of the data is below it<br>- Third quartile: when the data set is sorted in ascending order, 75% of the data is below it</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*TVti04AYDUCALwvmxGa1kQ.png" /><figcaption>Image source: Flatiron Data Science Bootcamp curriculum</figcaption></figure><p>The Seaborn documentation for the box plots I use below can be found <a href="https://seaborn.pydata.org/generated/seaborn.boxplot.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCHUD6mvP1_up9OsThHSPw.png" /></figure><p>As the box plots of price vs. year built don’t show any discernible patterns or trends as we go from year 1900 to 2015, visualizing the data in this way led me to conclude that there was likely not a strong relationship between these two variables. However, looking at grade vs. price using box plots suggested that grade rating may be one of the strongest predictors of price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfkEi19NlLnE9pX5KqT1Xg.png" /></figure><p>Indeed, my final model excluded year built, but found grade rating to be the second most import factor in predicting the price of a home.</p><h4><strong>Violin Plots</strong></h4><p>A violin plot is similar to a box plot, but it incorporates a kernel density plot to allow for the visualization of the distribution of the data. In fact, the violin plot has a box plot inside it, so you get all the same information that a box plot provides while being able to understand the frequency of values. The “violin” part is the density plot turned on its side and you read it the same way that you’d read a density plot: thicker sections mean those values are more frequent in the data set.</p><p>View the Seaborn documentation for violin plots <a href="https://seaborn.pydata.org/generated/seaborn.violinplot.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhGqAp2EPNZz37gEatJ7Pw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyNmsOKkcT1cx2g8I0mEuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnWBMrIwvnkHlwAUai4jDw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fcQ8l93jlnKzIYrry7yxeQ.png" /></figure><p>The two violin plots displayed above helped me understand the limitations of a model built with this particular dataset. The dataset includes the most information about houses with 2–5 bedrooms and .75–3.5 bathrooms. The model won’t be very accurate when predicting the price of homes that don’t fall into these ranges.</p><h4><strong>Joint Plot</strong></h4><p>The joint plot from Seaborn allows us to visualize a scatterplot, distributions, kernel density estimation (KDE), and a simple regression line of two variables all in one visualization! For me, this single visualization ended up being the most useful during EDA. Joint plots provide so much information about both the predictor and target and helps check the linearity assumption between the two, which is necessary for regression analysis<br><br>The full documentation for Seaborn’s jointplot can be found <a href="https://seaborn.pydata.org/generated/seaborn.jointplot.html">here</a>.</p><p>Here are the joint plots for what ended up to be my top 5 features for predicting the price of a home. In each you can see the positive regression line between the predictor and price (target).</p><p>#1 Size — Square Footage of Living Space</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84F9oGdFvfYSBi8dpAuA5g.png" /></figure><p>#2 Quality — Grade Rating</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ea_Vh6oyIC0q_LbFCiBjpQ.png" /></figure><p>#3 Size — Square Footage of Lot</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIkdX8-ZsOk0iinCtEbdbg.png" /></figure><p>#4 Neighborhood — Square Footage of Living Space of 15 Nearest Neighbors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pD92KJjU8iJ259F7mD2Bzg.png" /></figure><p>#5 Features — Number of Bathrooms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9IrjSVUloiBmWL4gxPGQw.png" /></figure><h3><strong>Heatmaps</strong></h3><p>Heatmaps build a color-coded correlation matrix that displays a table of the correlation coefficient between two variables. The correlation coefficient can range from -1 (a perfect negative correlation) to +1 (a perfect positive correlation).</p><p>I used this type of visualization to examine possible multicollinearity between potential features in my model. As the goal of regression analysis is to understand the relationship between each predictor and the outcome, correlation between predictors can be a problem because there is no way to change one predictor while holding all others constant. With correlation, changes in one predictor will be associated with changes in another predictor as well. This can impact the coefficients and statistical significance of the features in our model, making accurate interpretation difficult.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sf-G9HexPESB9K4EeaNtIg.png" /></figure><p>The Seaborn annotated heatmap was particularly useful. Here’s the <a href="https://seaborn.pydata.org/examples/heatmap_annotation.html">documentation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2ccRBpbcIyG1ihDTdvyPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*i_7l4t38T9pYX4YV-eCRVg.png" /></figure><p>The heatmaps helped me determine that I could not use both sqft_living and sqft_above in my model since they are highly correlated with one another (correlation coefficient = 0.877). Ultimately, I decided to only include sqft_living in the model.</p><h4>Returning to my original goals for EDA visualization, here’s what I learned:</h4><blockquote><strong>1. Assess the need for additional Data Cleaning</strong></blockquote><blockquote>Visualization helped me decide to drop the Waterfront variable entirely.</blockquote><blockquote><strong>2. Determine Potential Features for the Model</strong></blockquote><blockquote>Visualization led me to believe that sqft_living, grade, sqft_lot, saft_living15, and, bathrooms all seemed like very promising features to predict house price.</blockquote><blockquote><strong>3. Discover Potential Limitations of the Model</strong></blockquote><blockquote>Based on our dataset, the model won’t handle outliers particularly well. As we saw in our violin plots, the model will be best suitable to predict the price of houses that have 2–5 bedrooms and .75–3.5 bathrooms.</blockquote><blockquote><strong>4. Evaluate Multicollinearity Between Predictors</strong></blockquote><blockquote>Sqft_above and sqft_living had the highest correlation coefficient (0.877). I couldn’t keep both in my model. I decided to only keep sqft_living, as it seemed that it may be one of the strongest predictors based on its joint plot with price. I ultimately dropped sqft_above.</blockquote><p><strong>Now I’m ready to move on to feature engineering and modeling!</strong></p><p>While visualization can be a fun and beautiful part of EDA, it’s also incredibly useful. As a very visual person, I gain some of the most insight from my data by visualizing it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fafcf6ecdadc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>