<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Britton on Medium]]></title>
        <description><![CDATA[Stories by Matt Britton on Medium]]></description>
        <link>https://medium.com/@matthew.james.britton?source=rss-6e9905adba83------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*XjxEtGRRr8foVrtrWWqD5Q.jpeg</url>
            <title>Stories by Matt Britton on Medium</title>
            <link>https://medium.com/@matthew.james.britton?source=rss-6e9905adba83------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:09:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matthew.james.britton/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Because I Forget Every #@$&*! Time]]></title>
            <link>https://medium.datadriveninvestor.com/because-i-forget-every-time-73a49874da98?source=rss-6e9905adba83------2</link>
            <guid isPermaLink="false">https://medium.com/p/73a49874da98</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Matt Britton]]></dc:creator>
            <pubDate>Thu, 12 Nov 2020 23:30:20 GMT</pubDate>
            <atom:updated>2021-04-06T06:53:45.538Z</atom:updated>
            <content:encoded><![CDATA[<h3>Because I Forget Every Time</h3><h4>A Brief Guide to the Confusion Matrix</h4><p>You know that acquaintance whose name you always forget? And then you see them on the street, and have to figure out whether your relationship to them, and your own personal level of coolness, can support a “<em>Hey you!</em>”, a “<em>Hey friend!”</em>, or even a “<em>Weellll, look what the cat dragged in!</em>”.</p><blockquote>I apologize to anyone reading this who I’ve greeted in this way before.</blockquote><p>Alas, there are three dear friends whose names I’ll never remember, and it’s a source of deep shame. Precision, Recall, Specificity. All the mnemonics, headcanons, and rote repetition have been for naught — I simply never remember which is which. I’ll even admit to sometimes getting flummoxed by True Positive Rate.</p><p>Maybe this is not an affliction you suffer from. In which case, have a chuckle and pass on by. But, we all know the confusion matrix is called so for a reason. If you’re anything like me, you’ll want to bookmark the chart below and begin the long, slow process of sounding like you understand your own model in work meetings.</p><figure><img alt="Chart showing graphical depictions of various scores related to the confusion matrix: e.g. accuracy." src="https://cdn-images-1.medium.com/max/992/1*hf6QV3Ush8xYLmWwsIB3eA.png" /><figcaption>If you’re like me, you’ll want to bookmark this so you can refer to it at least once a week.</figcaption></figure><p>Please let me know if I made any mistakes, or if you’d like to see additional scores added to this chart.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73a49874da98" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/because-i-forget-every-time-73a49874da98">Because I Forget Every #@$&amp;*! Time</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meal Planning the Data Visualization Way]]></title>
            <link>https://medium.com/data-science/meal-planning-the-data-visualization-way-83dc8cf91d96?source=rss-6e9905adba83------2</link>
            <guid isPermaLink="false">https://medium.com/p/83dc8cf91d96</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[recipe]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cooking]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Matt Britton]]></dc:creator>
            <pubDate>Sun, 02 Aug 2020 05:40:25 GMT</pubDate>
            <atom:updated>2020-08-02T13:47:19.862Z</atom:updated>
            <content:encoded><![CDATA[<h4>A simple app to help you plan a week’s worth of meals, with a focus on reducing waste and simplifying shopping trips.</h4><figure><img alt="People eating at a table." src="https://cdn-images-1.medium.com/max/1024/1*XitNX6spa0g4Z_4PtNOs5A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@spencerdavis?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Spencer Davis</a> on <a href="https://unsplash.com/s/photos/meal?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>. All other images by author where not otherwise specified.</figcaption></figure><p>Sunday is usually my family’s grocery shopping day, and so Sunday early-afternoon usually sounds a bit like this:</p><blockquote>Can we make pizza at least twice?</blockquote><blockquote>I’m getting tired of beans…</blockquote><blockquote>We should make beans and rice! And curry. And order sushi!</blockquote><blockquote>That’s too much rice…</blockquote><p>Actually picking food to make for a whole week is a balancing act between everyone’s preferences, our budget, and a bunch of opinions about what constitutes healthy choices. And, we would also like grocery shopping to be easy, and to not have to buy a large quantity of something that will go to waste (looking at you, bunches of coriander).</p><figure><img alt="Slightly humorous photo of coriander." src="https://cdn-images-1.medium.com/max/1024/1*xaBNuflQdZPz6ImI7jgQBQ.jpeg" /><figcaption>The guilty party. Photo by <a href="https://unsplash.com/@surojadek?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tomasz Olszewski</a> on <a href="https://unsplash.com/s/photos/coriander?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>.</figcaption></figure><p>So, I built a small <a href="https://mybinder.org/v2/gh/MattJBritton/MealPlanner/master?urlpath=lab/tree/app.ipynb">app</a> that generates meal plan suggestions and visualizes the overlap between ingredients. Because you upload your own data file of recipes, you can choose how to categorize ingredients, and therefore the app can help optimize for whatever is important to you. For example:</p><ul><li>Save money by buying a large quantity of one ingredient that will be used in multiple recipes</li><li>Reduce food waste</li><li>Pick meals with varied types of grains, veggies, spices, etc to appease everyone</li><li>If you are vegetarian or vegan, get a balance of different types of protein</li></ul><h4>Visualizing a Meal Plan</h4><p>The tech here is not complicated. Basically, the app takes a file with a list of recipes, associated metadata (like the URL) and a list of ingredients by category. It looks like this:</p><figure><img alt="Sample of recipe box file." src="https://cdn-images-1.medium.com/max/1024/1*1p7KUQAbdY_7Ef7tAAv9og.png" /><figcaption>Sample of our “recipe box” file.</figcaption></figure><p>You pick an initial recipe of interest to get started, and then the algorithm adds new recipes to the meal plan until you have 5 (or however many you want). The next recipe is selected by <em>intersection over union</em> — the count of ingredients that a recipe shares with all ingredients in the meal plan so far, divided by the count of all ingredients. This formula is then randomized slightly so we get recipes that share something in common, but not ten different versions of paella.</p><p>Intersection over union actually maps pretty closely to my thought process for doing meal planning in the dark old pre-app days. So, I designed the output to visualize which ingredients are common or unique to each recipe.</p><figure><img alt="An example meal plan visualized." src="https://cdn-images-1.medium.com/max/1024/1*94H0e7ZgOWe6_QJT9VTZSw.png" /><figcaption>An example of a meal plan.</figcaption></figure><p>Ingredients are colored by the “category” specified in the recipe file. This makes it really easy to see whether we have a variety of proteins, or veggies, or grains, or whatever. You could conceivably set the categories by the grocery store where each item is cheapest/most regularly available, or by the categories used by a diet regimen.</p><p>Above I can see that two of the meals have rice, and two have noodles, and another one doesn’t have any carbs. Maybe I want to buy bread for that? One meal has no protein, so maybe I want to add something to the recipe. We use several veggies in multiple recipes, so it makes sense to buy in bulk. Scallions are also frequent offenders in the “too much per bundle and hard to freeze so they end up going bad category”, so it’s good that they are used in three recipes.</p><h4>The App</h4><p>The application is built on Jupyter Lab and uses Jupyter Widgets for controls.</p><figure><img alt="Search functionality of the app." src="https://cdn-images-1.medium.com/max/304/1*ewJsQ6v9qFkypvHq14ZzuA.gif" /></figure><p>You can search recipes by name and ingredient.</p><p>You can change the number of recipes per week and get new meal plan suggestions.</p><figure><img alt="Example of how to see new meal plans in the app." src="https://cdn-images-1.medium.com/max/1024/1*nHweJFCiJqnwO_HSoTFOew.gif" /></figure><p>You can save meal plans and return to them later.</p><figure><img alt="Example of how to save a meal plan in the tool." src="https://cdn-images-1.medium.com/max/1024/1*v8WQ-s49RFWAP5tRSDZEhA.gif" /></figure><p>Lastly, hovering over a recipe will show it’s name and metadata. We use this for the cookbook or URL where the recipe lives, but you could also add cuisine, number of calories or weight watchers points, etc.</p><p><strong>Getting Started</strong></p><p>You can access the app on <a href="https://mybinder.org/v2/gh/MattJBritton/MealPlanner/master?urlpath=lab/tree/app.ipynb">Binder</a>, or clone it from <a href="https://github.com/MattJBritton/MealPlanner">GitHub</a>. The only thing required is a file of recipes that you like to make. Ours has 300+ rows and took a couple hours of going through cookbooks to create. Alternatively, the app loads by default with a sample file of recipes pulled from the <a href="https://spoonacular.com/food-api">Spoonacular API</a>. You can get as many recipes as you want using the <em>get_sample_recipes.py</em> script, although note that the free tier has a daily limit of several hundred. You’ll need a free API key to do this. This is a good option if you want to explore new cuisines and foods, as Spoonacular’s recipes are drawn from all kinds of food websites and blogs.</p><blockquote>Please let me know in the comments if this tool is useful to you!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83dc8cf91d96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/meal-planning-the-data-visualization-way-83dc8cf91d96">Meal Planning the Data Visualization Way</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Escape the Correlation Matrix into… Feature Space.]]></title>
            <link>https://medium.com/data-science/escape-the-correlation-matrix-into-feature-space-4d71c51f25e5?source=rss-6e9905adba83------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d71c51f25e5</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <dc:creator><![CDATA[Matt Britton]]></dc:creator>
            <pubDate>Mon, 25 May 2020 14:13:30 GMT</pubDate>
            <atom:updated>2020-05-27T04:05:35.968Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-sa/4.0/</cc:license>
            <content:encoded><![CDATA[<h4>Introducing a new technique for plotting correlation matrices. It enables you to quickly identify high-level structures in your data.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d-dmvcXbAcBjJYWfzevm-A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@juskteez?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Juskteez Vu</a> on <a href="https://unsplash.com/s/photos/space?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>. All other images are by the author.</figcaption></figure><p>We’ve all had the experience of getting good at using a bad product. The wonky CRM your company is locked into a four-year contract with. The form tool with the hideous database connector that you never find time to migrate away from. MS Access. Due to inertia, organizational limitations, or cost, we often find ourselves using subpar tools and technologies, learning their little quirks, devoting muscle memory to a repetitive series of dialogue boxes. Have you ever tried to save an Excel file as a CSV?</p><blockquote>Some features in your workbook might be lost if you save it as a CSV.</blockquote><p>Sometimes, using these tools is a necessity. Who knows, maybe it even builds character?</p><blockquote><em>But as data scientists, we often have the ability to choose our own tools. And doing so is very important, because our tools shape our sense of what is easy, and what is even possible at all.</em></blockquote><p>The correlation matrix is one of these bad tools. To be more specific, it is a tool that is effective for a limited set of tasks, in a situation where we habitually need to perform a broad array of tasks. And because the correlation matrix has a long history and works just well enough on smaller datasets, we treat it like the only game in town. I present the Feature Space Diagram — a tool that complements the correlation matrix and is useful for both data scientists and non-experts. The code is easy to adapt for yourself — see the <a href="https://gist.github.com/MattJBritton/b6944218903312f6220bfb48706b593c">minimal example</a> and the <a href="https://github.com/MattJBritton/ExquisiteCorrpse">full notebook</a>.</p><h4>How We Use The Correlation Matrix</h4><p>So what exactly is the issue with the correlation matrix? Let’s run through an example to see for ourselves. Say we are working with the <a href="http://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html">Boston Housing data set</a>, and our goal is to build a predictive model for the median house price in a neighborhood.</p><p>We might ask about missing values, look at univariate distributions, consult our data dictionary, but one of our biggest questions will be:</p><blockquote>how do the features relate to each other, and the target?</blockquote><p>As good data scientists, rather than just throwing a neural network at it, we want to understand our data set to the extent possible. This will help us with:</p><ul><li>Feature Engineering</li><li>Feature Selection</li><li>Dimensionality reduction / addressing multicollinearity</li><li>And down the line, communicating results to stakeholders</li></ul><p>So, we leap into action (figuratively, my <a href="https://xkcd.com/1329/">running desk</a> hasn’t come in yet), and build a correlation matrix. It calculates the <a href="https://en.wikipedia.org/wiki/Pearson_correlation_coefficient">Pearson Correlation Coefficient</a> (abbreviated <em>r</em>), with each pair of quantitative features in the data receiving a single number from -1 to 0 to +1, indicating that their linear dependence is strongly negative, absent, or strongly positive, respectively. When output as a table, it shows the strength of the (linear) relationship between any two features. We engage in some light visualization, and presto!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*jKlY7iYuI0XoHSx0M9UCMQ.png" /></figure><blockquote>But how do we use this chart? Let’s ask ourselves…</blockquote><blockquote>— What do we want to do?</blockquote><blockquote>— Does this thing do it for us?</blockquote><p>As data scientists, we want to do at least the following:</p><ol><li>Find the correlation between any two features of interest.</li><li>Get a sense of the high-level structure of the data — is there a lot of multicollinearity? Where are the “clusters” of closely related features?</li><li>Learn something we don’t know, and might not think to ask.</li></ol><p>The correlation matrix above clearly excels at the first point. It’s essentially just a look up table, improved by double-encoding the values with color.</p><p>The second point is a little trickier. “Structure” here means forms of organization that exist among aggregates of individual elements — like looking at a house and saying “it has 3 floors” or “the foundation is sinking” rather than just “those two bricks are adjacent”. In feature space, the structures we are looking for are clusters of collinear features or independent features that offer unique predictive value.</p><p>More complex relationships are possible in high-dimensional space , which points us towards the third objective — learn something we don’t know. If we were simply looking to validate a hypothesis, there are plenty of statistical tests. But getting an insight that we had not even thought to consider is the heart of analysis and visualization.</p><h4>Too Corre-little, Too Correlate</h4><blockquote>So, does our correlation matrix help us find clusters of features, and generate novel insights about the data?</blockquote><p>Kinda? We can tell from the prevalence of dark squares whether we have a lot of collinearity. We can look into a row or column and notice that it is very dark or very light, and get a sense of whether that feature is close to many other features or not. But, it is going to take a lot of time staring at this to identify clusters. One issue is that all of the columns look a little bit the same, and we have to keep checking the axis labels to make sure we are looking at the right one.</p><p>With time and effort, we can deduce that, for example, <em>RAD</em>, <em>TAX</em>, <em>INDUS</em>, <em>AGE</em>, and <em>NOX</em> are highly correlated with each other, and might form a cluster. The information necessary to do that is all here.</p><blockquote>But the purpose of a visualization is to leverage our evolution-honed visual acuity to quickly spot patterns that are not visible when just staring at a table of figures.</blockquote><p>On that count, I think this fails as a tool for generating insight.</p><p>But wait! If clusters are what we are after, then why not just sort the axes so that related features are next to each other?¹ Can you see the improvement from the original version to this one?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*jKlY7iYuI0XoHSx0M9UCMQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*ZTEDkqqyJKChoEWDKtVvjg.png" /><figcaption>Unsorted (left) and sorted (right) correlation matrices. Feature clusters are easier to pick out in the sorted version.</figcaption></figure><p>Now the rubber hits the road! Just scan along the diagonal and look for blocks of color. We can immediately see clusters between <em>CRIM</em>, <em>RAD</em>, and <em>TAX</em>, and <em>INDUS</em>, <em>AGE</em>, <em>NOX</em>, and <em>DIS</em>.</p><p>What we’ve done here is leverage our brain’s evolution-honed <a href="https://www.csc2.ncsu.edu/faculty/healey/PP/">preattentive processing</a> to quickly pick out patterns based on <a href="https://www.usertesting.com/blog/gestalt-principles">Gestalt principles</a> such as proximity and similarity.</p><blockquote>One of the main goals of visualization is to bind interesting data to these patterns — we are basically building binoculars and asking our brain to tell us what’s moving in the forest.</blockquote><p>In addition to these clusters, our brain picks out other patterns. Sure <em>RAD</em> and <em>TAX</em> have a high correlation, but they are both correlated to a lesser extent with <em>CRIM</em> and <em>B</em>. <em>LSTAT</em> seems like it’s in a cluster with <em>Median_Home_Value</em> and RM, but it has reasonably strong correlation with most of the other features. It also looks like the <em>NOX</em>/<em>DIS</em> and <em>RAD</em>/<em>TAX</em> clusters are correlated with each other, but not as much with the <em>LSTAT</em>/<em>RM</em> cluster.</p><h4>Making the Most of the 2nd Dimension</h4><p>These are the complex, surprising structures we talked about earlier. Our sorted correlation matrix makes them visible, but we still need to reference the axis labels and trace down a row or column, and for a data set with hundreds of columns the correlation matrix won’t even fit on a page!</p><p>Another limitation of the correlation matrix is that both axes are sorted in the same order, and so a feature can only be directly adjacent to 2 others. This constrains our ability to use proximity as a measure of closeness. In the sorted matrix, we are trying to analyze relationships which are encoded well with color, but not with distance. This gives the chart a sort of “magic eye” character, and makes it less approachable for non-experts.</p><blockquote>Hmm, what chart lets us position elements in 2 dimensions, so we can have horizontal and vertical proximity?</blockquote><blockquote>A scatterplot!</blockquote><p>All we need is a way to generate X,Y coordinates for each feature. The full method is available in the <a href="https://github.com/MattJBritton/ExquisiteCorrpse">source code</a>, and is summarized below:</p><ul><li>Take the absolute value of our correlation matrix, and subtract each value from 1. It is handily transformed into a distance matrix!</li><li>We can then use PCA to reduce our NxN matrix to Nx2.</li><li>Plot each feature’s location using the two principal components.</li><li>Use Feature Agglomeration to generate feature clusters.</li><li>Color each feature by its cluster.</li><li>Draw lines to represent relationships of at least <em>r</em> = 0.7.</li></ul><h4>Feature Space Diagrams</h4><p>The Feature Space Diagram (FSD) below uses a combination of position, color (for cluster membership) and linkage (for strong correlations) to describe relationships between features in complementary ways. Even at a glance, we can see that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C88xSabhA5MuZOYdcF2nlw.png" /><figcaption>Feature Space Diagram for the Boston Housing data set.</figcaption></figure><ul><li><em>CHAS</em> is totally independent².</li><li>Most features are lumped into a large cluster, but within that a core set are strongly correlated.</li><li>Compared to the correlation matrix, we are able to quickly identify that <em>LSTAT</em> occupies a unique place in being correlated with the target AND many other features, including <em>RM</em>, which is otherwise independent.</li></ul><blockquote>This is a quick insight that could be very helpful in feature engineering and selection. If we need the details, we can look them up in our correlation matrix, putting it into its rightful place as a lookup tool.</blockquote><h4>Scaling Up</h4><p>The Boston Housing data is pretty small, so the correlation matrix and the feature space diagram are more evenly matched. But the FSD really shines with larger data sets. Let’s take a look at the FSD for the <a href="https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)">Breast Cancer data set</a>, which has 30 columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0R4jPjeugom8Uu9EgzhJA.png" /><figcaption>Feature Space Diagram for the Breast Cancer data set</figcaption></figure><p>We can immediately see that there are distinct clusters of features in this data. The orange and brown clusters are strongly correlated internally and with each other. The features in the red cluster are both further from the orange cluster, and less strongly correlated with each other. There are also outliers too such <em>smoothness_worst, area_worst</em> and <em>concavity_mean</em>. <em>Radius_sd_error</em> and <em>concave_points_mean</em> are strangely related to each other and nothing else.</p><p>This data consists of several types of measurements (radius, perimeter, symmetry, etc) and statistics computed for each of these measurements (mean, standard deviation, worst). One high-level question we could try to answer is whether we can just use the “mean” of each measurement type and toss out the rest. In other words, do features cluster by measurement type?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aj_KCY5O2swQiaiW9ZGXA.png" /><figcaption>Feature Space Diagram with user-provided colors mapping domain knowledge</figcaption></figure><p>By hard-coding a color mapping of feature to measurement type, we can easily see the answer!</p><blockquote>Nope!</blockquote><p>There are a few measurement types that are physically adjacent, such as <em>compactness</em>_ in orange. But overall, measurement type doesn’t drive correlation here.</p><p>Still, this is a good hypothesis to reject. This approach would also be very helpful if we had merged several data sets together for our analysis, and now wanted to identify to what extent their features overlap. Just give the variables from each data set their own color.</p><h4>Time Series</h4><p>I was particularly surprised by the FSD created for a time series data set on <a href="https://archive.ics.uci.edu/ml/datasets/Epileptic+Seizure+Recognition">Epileptic Seizure Recognition</a>. Each data point consists of 178 EEG measurements taken over the course of a second. One of the first questions we want to ask of a time series is whether there is any repetitive or cyclical behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60788CnaeGXZWQOnlQFdUA.png" /><figcaption>Feature Space Diagram for Epileptic Seizure Recognition data set</figcaption></figure><p>This vaguely Australian-looking blob shows us that each feature is only correlated with its neighbors (i.e. the preceding and following time slices). The odd shape comes from PCA trying to push each feature away from others based on chronological order.</p><p>However, how do we explain all the bends and curlicues? We might reasonably expect a Markov chain to look something more like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJhvF9_P_WmhKYvxgd5oUA.png" /><figcaption>Feature Space Diagram for synthetic semi-random Markov Chain</figcaption></figure><p>In other words, a circular shape without loops. This is a hint that there are other correlations in this data set, aside from between each chunk of 5 consecutive timestamps. Let’s check our correlation matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LmKAAPnzVXLz3CghKuhLQ.png" /></figure><p>At first glance, the correlation matrix supplies the same story — a band of highly correlated points extending 2–3 timestamps before and after each point, and no correlations otherwise. But if we look closely, we can see two sets of faint lines parallel to the main line — one set about 17 points before and after each point, and the other about 35 points before and after each point. This odd visual artifact suggests that there is a small cyclical effect in this data. In the FSD, we can even see that the line bends so that X18, X33, and X49 are as close as possible together, approximating the trend we see above. Pretty cool!</p><h4>Conclusion</h4><p>The Feature Space Diagram is a tool that complements the correlation matrix and provides a faster synopsis of correlations between quantitative features. In this article, I show how it can be used to surface interesting relationships which require further investigation. The code is available below (or see the full notebook <a href="https://github.com/MattJBritton/ExquisiteCorrpse">here</a>) and includes additional configurations and interactive features, including:</p><ul><li>Zoom and pan</li><li>A widget to select the number of clusters</li><li>A widget to change the <em>r</em> threshold for showing correlation lines</li><li>A chart showing the number and size of clusters</li><li>Ability to pass in user-defined coloring schemes, as shown above</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EM3qWADKRKwSCLXFbHkLgA.png" /><figcaption>Full interactive version of the Feature Space Diagram</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af9dc06e2591d92c21cd240fb61ad701/href">https://medium.com/media/af9dc06e2591d92c21cd240fb61ad701/href</a></iframe><h4>Footnotes</h4><p>¹ For a full implementation of this sorting, see the <a href="https://rdrr.io/cran/lessR/man/corReorder.html">corReorder R package</a>.<br>² In the examples for this article, the CHAS boolean variable is included in correlation matrices. Although it is coded as a number, the Pearson Correlation Coefficient is not designed to handle categorical variables, and so this should not be done in practice. See the <a href="https://towardsdatascience.com/rip-correlation-introducing-the-predictive-power-score-3d90808b9598">Predictive Power Score</a> for a similar feature score that can handle variables of any type.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d71c51f25e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/escape-the-correlation-matrix-into-feature-space-4d71c51f25e5">Escape the Correlation Matrix into… Feature Space.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Trees from the Root Up]]></title>
            <link>https://medium.com/data-science/decision-trees-from-the-root-up-1f169b6e0460?source=rss-6e9905adba83------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f169b6e0460</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Matt Britton]]></dc:creator>
            <pubDate>Sun, 26 Apr 2020 02:00:57 GMT</pubDate>
            <atom:updated>2020-06-16T18:29:44.650Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build an optimal decision tree by hand to understand the surprisingly common-sense mechanics of this ML stalwart.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOUyNsQd2_N189m7_4fc6A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@leliejens?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jens Lelie</a> on <a href="https://www.unsplash.com/">Unsplash</a></figcaption></figure><p>Decision trees are one of the foundational model types in data science. And luckily, they provide a great example of how computers can automate simple human intuitions to build large, complex models. In fact, decision trees optimize a simple criterion, which mirrors how we make decisions in everyday life. Understanding how predictive models mirror our own mental models allows us to think critically about how they might go wrong, and what a good model looks like.</p><p><strong>Overview of this tutorial</strong></p><p>The goal of this article is to provide an interactive introduction to the theory of decision trees. By the end of this tutorial, you should be able to:</p><ul><li>Describe the structure and function of a decision tree.</li><li>Construct a small decision tree by hand using the concepts of entropy and information gain.</li><li>Reason about how algorithmic implementations build decision trees at scale.</li></ul><p><strong>Accessing the code for this tutorial</strong></p><p>The code and visualizations in this article were all generated in Python using numpy, pandas, and Altair (images by the author except where noted). You can follow along and interact with the charts by using the methods below. Since the purpose is to build your own decision tree by hand, this is highly recommended!</p><ul><li>Method 1: <a href="https://github.com/MattJBritton/InteractiveDecisionTrees">Github</a>. Clone or fork the repository and install the environment via the YAML file at binder/environment.yml.</li><li>Method 2: <a href="https://nbviewer.jupyter.org/github/MattJBritton/InteractiveDecisionTrees/blob/master/InteractiveDecisionTrees.ipynb">NBViewer</a>. Access a version of the notebook with interactive charts.</li><li>Method 3: <a href="https://mybinder.org/v2/gh/MattJBritton/InteractiveDecisionTrees/master?filepath=InteractiveDecisionTrees.ipynb">Binder</a>. Access a complete live version of the notebook online. You can modify and run the notebook, or even try it with another data set.</li><li>Method 4: Follow the links placed in-line throughout the article to view the interactive charts on Datapane.</li></ul><h3><strong>What is a Decision Tree?</strong></h3><p>A decision tree is a logical model that helps you make a prediction based on known data. This prediction consists of whether or not something will happen, or whether an item belongs in a category or not. For example, predict whether a tumor is malignant or benign based on a patient’s medical history.</p><p>This prediction is made using a series of Yes or No questions. “<em>Is the patient a smoker?</em>” is a question that helps a decision tree divide the data into subsets, each with their own behavior. If this behavior is easier to predict than for the whole data set, then this decision tree is effective. Let’s make this concrete with an everyday example.</p><p><strong>Everyday Examples of Decision Trees</strong></p><p>You’ve undoubtedly made a decision tree at some point in your life. Let’s take the example of packing for a vacation. Should you bring a bathing suit? Your decision process might look a little like this:</p><figure><img alt="Decision tree with questions specified for each node, and labeled arrows for terminology." src="https://cdn-images-1.medium.com/max/768/1*_hFj-jCys9RcszgprXrJUQ.png" /><figcaption>The grey represents the actual structure of the tree, while the purple arrows indicate key terminology.</figcaption></figure><p>Decision trees always begin at a single node (the top grey bubble), representing the first question to ask. In this example, we ask if our destination has a place to swim. This question is referred to as a split¹. It must be Yes/No — note that we handle the temperature quantitative scale by creating a threshold at eighty degrees.</p><p>Eventually, we get to a “leaf” node. This is where we make our decision. Also note that each node is a “parent” of two “child” nodes unless it is a leaf.</p><h3><strong>Building Good Decision Trees</strong></h3><p>This model makes intuitive sense to us. Remember, though, that in machine learning we have to evaluate models, and we’ll often be working with complex data sets for which we have limited intuition. So how might we evaluate the model above? More broadly, what characteristics differentiate a good model from a bad one? A couple of examples will help us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*bRpaqAeAdAjRQ-aX8uHlqg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*qvL-SpG4sjwhK4nsta7bEg.png" /><figcaption>A good decision tree that produces relatively “pure” child nodes, and a bad decision tree that perpetuates uncertainty by asking an irrelevant question.</figcaption></figure><p>On the left, we have an example of a good decision tree. One way we can assess the usefulness of this mental model is to ask ourselves how it would have performed on our last 10 trips, represented here by colored squares. Red for a trip where we didn’t need a swim suit, and blue for ones where we did.</p><p>We can see that we initially have a 60/40 split, but that our “No” branch is all red. In other words, in 3 out of our last 10 trips, we knew beforehand that there was no place to swim, and on all of those trips, we did not regret our lack of bathing suit. So when we answer “No” to this question, we can be very sure we are making the right decision.</p><p>In 7 out of our last 10 trips, we were aware of a place to swim, and in 6 out of 7 of those occasions, this was the right choice. What happened on the last one? Maybe it was too cold, or the pool was closed unexpectedly. The good thing is that for both of our branches, we are able to make a better guess than we could if we did not ask the question.</p><p><strong>Bad decision trees</strong></p><p>On the right, we have a question that doesn’t seem to have anything to do with swim suits. Moreover, we can see that our question subdivides our past trips into two subsets that each have the same Bring It/Leave it at Home split. If both the Yes/No branches have the same distribution (60/40), then asking the question didn’t give us any new information. So why did we bother asking the question in the first place?</p><p><strong>Defining a “good” decision tree</strong></p><p>We can run with this observation and provisionally state that our goal in building decision trees is to produce “pure” child nodes with as few questions as possible. And it follows that if this is the goal of the tree, then each split should aim to “purify” its child nodes to the extent possible.</p><p>Hold on to this concept of “purity”, as it will be useful in a second. In the meantime, let’s switch gears to a real data science problem.</p><h4>Building a Decision Tree by Hand on the Titanic Data Set</h4><p>For our real problem, we’re going to use the Titanic data set². Each row is a passenger along with demographic and travel information such as their age, sex, and the class of their ticket. The usual purpose is to predict whether a passenger survived or perished.</p><figure><img alt="First 5 rows of the data set, with columns for Survived, Sex, Age, Port of Embarkment, Family Size, and ticket Class." src="https://cdn-images-1.medium.com/max/874/1*ePshx2hTNA8FAq7Deu9EEg.png" /><figcaption>First 5 rows of the Titanic data set.</figcaption></figure><p>How would we go about building a decision tree to predict which passengers survived? What’s a good first split?</p><p>Without any statistics or visual instruments, all we have to go on is any existing knowledge about this data set or the domain. Maybe we’ve seen the movie and guess that more women survived than men. Or we can use knowledge about the world and the design of the ship to guess that first class passengers fared better than third class.</p><p>Of course, as data scientists, we will often be called on to build models on data from new domains, or about which we have little prior intuition. In other words, <strong>we need a data-driven way to choose good splits</strong>. We can start with the distribution of each feature across each class (Survived and Perished). We’ll use stacked bar charts for categorical features, and <a href="https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0">KDE</a> (kernel density estimate, basically a smoothed histogram) plots for quantitative features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ozhPyIaXMtxhYPkcrYqIVQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LDR8TAngplt43b5Hr7cjEg.png" /><figcaption>Distributions for Survived and Perished passengers across 5 features.</figcaption></figure><p>How do we use distributions to identify candidate splits? When looking at categorical variables, we want to select a category that has a markedly different distribution than the others. For example, we can see that men and women survived at different rates. For quantitative variables, we’re hunting for a region where one line is much higher than the other. <em>Age &lt; 10</em> seems to fit that bill, as do individuals traveling alone (<em>Family Size &lt; 1</em>).</p><p>But now that we have several potential splits, how do we evaluate whether they are any good? How do we compare multiple options?</p><p>Remember the “pure child nodes” objective we discussed earlier? We can use this concept to build a quantitative measure of the “goodness” of a split.</p><h3>Entropy and Information Gain</h3><p>We’re going to define <em>entropy</em>, a measure of the purity of a node, and <em>information gain</em>, a measure of how much a split improves the<em> entropy</em> from a parent node to its children³. Below you’ll find the definitions and formulas. If this is a bit confusing, feel free to skip to the visual below which presents the metrics in context.</p><p><strong>Entropy</strong></p><ul><li>A measure of how “pure” a node is.</li><li>Based on the ratio of the two classes, e.g. survivors to non-survivors.</li><li>Ranges from 0 to 1. 0 means a pure node (all survivors or vice versa) and 1 means a 50/50 split. Note that an entropy of 1 corresponds to the “worst-case scenario” where we are least able to make an educated guess.</li></ul><p><strong>Information Gain</strong></p><ul><li>A measure of how much a split decreases entropy (a bigger decrease is better, as 0 is the goal).</li><li>Equal to the entropy of the parent node minus the weighted average of the entropies of the child nodes.</li><li>The algorithm that builds decision trees finds the split at each node having the highest information gain.</li></ul><figure><img alt="Mathematical formulas for entropy and information gain" src="https://cdn-images-1.medium.com/max/1024/1*Xtnvia4phm57X-_pWkSHPw.png" /><figcaption>Formulas for entropy and information gain</figcaption></figure><h4>Visualizing the Entropy of Parent and Child Nodes</h4><p>We can get a better intuition for how to choose the best splits by overlaying the calculations on the decision tree structure. The visual below evaluates the information gain of the split <em>Class = Third Class</em>.</p><figure><img alt="Diagram showing the information gain calculation from entropy of parent and child nodes." src="https://cdn-images-1.medium.com/max/619/1*5ZpYLdE9t0X6Gi_7se_d6Q.png" /><figcaption>The Information Gain calculation for the split “Class = Third”.</figcaption></figure><p>The orange and blues squares show the breakdown of survived/perished at each node. For each node, we also have the entropy, survival rate, and the number of people. Together, these facts tell a story about the data set AND allow us to evaluate how good this split might be:</p><ul><li>Our data set as a whole consists of 891 people, 38% of whom survived. Because a 38/62 split is pretty close to 50/50, our entropy is almost 1.</li><li>491 passengers (55%) were 3rd class. These folks only had a 24% survival rate, much worse than overall. For this subset, the entropy thus drops to 0.8.</li><li>In the right branch, we have 1st and 2nd class passengers, who were more likely to survive. This subset had a 56% survival rate. However, even though we’ve learned something important about this group, the entropy here is actually a bit higher than in the overall data set. A 56% survival rate is even closer to 50/50 than the population 40% survival rate — note that the direction does not matter in determining entropy.</li><li>Because one side of our split had a significantly lower entropy, and the other basically stayed the same, and the low-entropy subset is larger, this split does reduce the overall entropy. This is reflected in the information gain formula at the bottom.</li><li>The information gain for this split is 0.076. Note that this number has meaning as an absolute quantity (relative to the overall entropy of the target variable), but is most useful in comparison to other splits.</li></ul><h4>Building a Decision Tree by Hand</h4><p>Armed with the concept of information gain, we can now calculate the relative value of an individual split. An algorithm for building decision trees can evaluate many potential splits quickly to find the best one. To do this manually, we need a tool to assist us.</p><p>This interactive visualization lets us try out many potential splits. The feature distribution charts from earlier have been augmented with the ability to select a split, by clicking on a category in bar charts or just panning on quantitative features. When a split is selected, the sub-charts below lists the information gain and the entropy of the left and right subsets.</p><p>Note that entropy and information gain figures have been multiplied by 100 for readability. Otherwise, the example above can be recreated exactly.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F2494c592-ea45-402a-b1ab-6f6cb34ea2c7%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F2494c592-ea45-402a-b1ab-6f6cb34ea2c7%2F&amp;image=https%3A%2F%2Fdatapane.com%2Fstatic%2Fpython_report_illustration.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3434c6e3c773af1cd83b4b23e4eaaf2/href">https://medium.com/media/e3434c6e3c773af1cd83b4b23e4eaaf2/href</a></iframe><h4>Building the Next Layer of a Decision Tree</h4><p>There’s a clear winner in information gain here: Male/Female is by far the best split. However, given that we want a more complex, multi-variable model, we can repeat this process on the left and right side to determine the best split for each of these nodes. We now are building a model like the following:</p><figure><img alt="Template for decision tree of depth 2 on Titanic data set." src="https://cdn-images-1.medium.com/max/710/1*Qf5m0lF8KXzLTX0BhtOapA.png" /></figure><p>You can use the interactive visualization for the female and male subsets below. This version adds a line chart showing the information gain for all possible splits. Look for any interesting scenarios where multiple splits have similar information gain values.</p><p><strong>Female</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F7022fd28-f26a-4b41-aaac-5b8a57862d53%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F7022fd28-f26a-4b41-aaac-5b8a57862d53%2F&amp;image=https%3A%2F%2Fdatapane.com%2Fstatic%2Fpython_report_illustration.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/920cf3c1fa25001a11722fe1b4cead9c/href">https://medium.com/media/920cf3c1fa25001a11722fe1b4cead9c/href</a></iframe><p><strong>Male</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F8d0d97f4-6d67-4ef9-a624-55753095b5b6%2Fembed%2F&amp;display_name=Datapane&amp;url=https%3A%2F%2Fdatapane.com%2Fmbritton%2Freports%2F8d0d97f4-6d67-4ef9-a624-55753095b5b6%2F&amp;image=https%3A%2F%2Fdatapane.com%2Fstatic%2Fpython_report_illustration.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=datapane" width="800" height="625" frameborder="0" scrolling="no"><a href="https://medium.com/media/f54e315bc82a377584640798ec744a3c/href">https://medium.com/media/f54e315bc82a377584640798ec744a3c/href</a></iframe><h4>Comparing our Handmade Tree to Scikit-Learn’s Version</h4><p>Now we can compare this simple, handmade model with the version produced by an implementation of a decision tree algorithm, such as the one in scikit-learn. The following code performs basic data cleaning and feature engineering, then builds the model and outputs its structure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acfe05869f1a089b0c311b137d45c7f4/href">https://medium.com/media/acfe05869f1a089b0c311b137d45c7f4/href</a></iframe><figure><img alt="Text version of scikit-learn decision tree structure." src="https://cdn-images-1.medium.com/max/552/1*phG5OQcf6j9WX-WovDt17g.png" /><figcaption><em>Structure of the scikit-learn decision tree. Output is from export_text() method.</em></figcaption></figure><p>This model has a total information gain (from the first node to the leaves) of 0.312, and an accuracy of 0.796. Did you get the same splits?</p><h4>Conclusion</h4><p>We sometimes think of machine learning models as a fundamentally different type of intelligence from our own. However, in this case, at least, the process of model-building is rooted in the same type of common sense solutions we might bring to bear on a problem. Decision trees are built using simple iterative greedy maximization of <em>entropy</em>, a quantity that we have an intuition for.</p><p>Moreover, gaining familiarity with the tree-construction algorithm helps us as data scientists to understand and appreciate the trade-offs inherent in the models we can make with a few lines of code.</p><p>In my next article on decision trees, we will build on these concepts to analyze the structures of more complex decision trees, and understand how trees can <em>overfit </em>to a training set. Understanding the structural characteristics of good decision trees, and placing the different types of structures into a taxonomy, is a very helpful skill for a data scientist.</p><h4>Footnotes</h4><p>¹Alternate terms for a split include <em>rule </em>or<em> test node. </em>I use <em>split </em>here to draw attention to how the predicate splits the data set into subsets.</p><p>²The data set consists of the 891 passengers in the “train” subset on <a href="https://www.kaggle.com/c/titanic/data">Kaggle</a>. Missing data for the Age column has been imputed. Details are available in the <a href="https://github.com/MattJBritton/InteractiveDecisionTrees">source code</a>.</p><p>³You will often see <em>gini impurity </em>as an alternative criterion for splitting nodes. For example, scikit-learn uses this as the default criterion, with the ability to switch to information gain. While this math is slightly different, the conceptual considerations are the same, and in practice it doesn’t matter much which one is used.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f169b6e0460" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/decision-trees-from-the-root-up-1f169b6e0460">Decision Trees from the Root Up</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>