<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Hill on Medium]]></title>
        <description><![CDATA[Stories by Daniel Hill on Medium]]></description>
        <link>https://medium.com/@danielbhill?source=rss-3c9383cf6bc6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*VaObm6zslcdBY-gcfGJmcg.jpeg</url>
            <title>Stories by Daniel Hill on Medium</title>
            <link>https://medium.com/@danielbhill?source=rss-3c9383cf6bc6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:29:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielbhill/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your strategy should drive your data, not the other way round]]></title>
            <link>https://medium.com/data-science/your-strategy-should-drive-your-data-not-the-other-way-round-99ca9f2bd18e?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/99ca9f2bd18e</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[office-hours]]></category>
            <category><![CDATA[strategy]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Tue, 17 Nov 2020 13:36:49 GMT</pubDate>
            <atom:updated>2020-12-04T18:04:57.306Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/office-hours">Office Hours</a></h4><h4><em>How the philosophy of science interacts with data science practice and business strategy</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kFy9HQwl2L8yPK9Y" /><figcaption>Photo by <a href="https://unsplash.com/@hpzworkz?utm_source=medium&amp;utm_medium=referral">Hassan Pasha</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the recent themes of business books and articles is having a “data driven strategy”. But what does that actually mean? Will strategic thinking fall by the wayside as the machines take over? What is the right relationship between strategy, data, and science?</p><p>We’ll explore these questions by diving into a perhaps unexpected area: the philosophy of science. The question of “what is science?” and how it relates to other ways of thinking, and the problems we are, or should be, trying to solve is not a new question. Through exploring the philosophical arguments around science we can come to a better understanding of how science should be used not just in the laboratory, but in the organizations that now increasingly use scientific ideas and approaches as they’re swept up in the data revolution.</p><p>Before we get too far into the details though, let me introduce you to a personal hero of mine….</p><h3>Thomas Kuhn</h3><p>In 1962, Thomas Kuhn wrote “The structure of scientific revolutions”, turning the philosophy of science on its head. Previously, science had generally been thought to be a linear process away from ignorance and towards truth, progressing through a gradual process of experimentation and discovery. Philosophers like Hume and Popper had poked and refined the idea over time, but the basic idea of scientific progress remained.</p><p>Kuhn’s insight was that science didn’t linearly progress towards some universal truth. Rather science operated through <em>paradigms:</em> dominant theoretical frameworks that set the course of scientific work, think Newtonian Gravity or Einstein’s relativity. The radicalism of this insight was Kuhn’s insistence that one paradigm wasn’t better or more “true” than another, rather paradigms became dominant and accepted as they were <em>useful </em>at solving the particular problems scientists were interested in at that point in time.</p><p>This changed science from a practice that stood aloof from culture and subjective human interests to a practice that was intimately involved with these forces. Though individual scientific techniques are reasonably objective (test tubes, mass spectrometers, hypothesis tests, and the like) the problems scientists chose to investigate with these techniques are anything but, and it is the selection of problems that ultimately determine the theories and paradigms that are developed.</p><p>Kuhn described a paradigm life cycle where normal scientific work takes place within a given paradigm. Over time this paradigm would start to show errors and anomalies as scientists began to explore areas on the limit of the paradigm’s ability to solve problems effectively. This would lead to a breakdown of trust in the paradigm, leading to a crisis and eventual shift to a new paradigm that better solved this new set of problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tsBjPH-D3KQcfZSj" /><figcaption>Warning: paradigm demolition in progress (Photo by <a href="https://unsplash.com/@loran01?utm_source=medium&amp;utm_medium=referral">Dan Loran</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><h3>Paradigm shift</h3><p>These famous paradigm shifts have become well known to the point of cliché today, but the key idea behind them remains crucial and powerful: they are fundamentally creative acts and cannot be derived from the paradigm they replace.</p><p>Let’s take the shift from Newtonian to Einsteinian physics. Newton physics was the dominant paradigm from the late 1600s to the early 1900s because it helped people understand and solve the problems they were interested in: building engines, flying planes, rockets etc. Over time the problems changed as scientists became more interested in light, which caused Newtonian mechanics to wobble significantly until Einstein stepped in to save the day with a new paradigm.</p><p>Einstein&#39;s’ flash of genius, that the speed of light was constant and it was everything else that was relative, could not have been derived from any of the Newtonian experiments that preceded it. One could watch a million apples drop from a million trees, or collect data on a billion billiard ball collisions, without going “ah, it seems the speed of light is constant here”. This insight, this paradigm shift, was fundamentally a creative reimagining of how to do physics in a way that allowed scientists to solve the problems around light they were interested in at that time.</p><h3>Disruption</h3><p>The paradigm shift doesn’t just live within physics. Remember a paradigm is at its core a <em>dominant methodology for solving problems</em>, and problems aren’t limited to academic scientific practice. Any organization that is trying to solve a problem (which is most of them) will be subject to these kinds of revolutions in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*haWzKdTiAhT6Murq" /><figcaption>No human required (Photo by <a href="https://unsplash.com/@ajufaan?utm_source=medium&amp;utm_medium=referral">Mohamed Ajufaan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>Let’s take the world of travel, for example. When was the last time you used a high street travel agent to book your holiday? I’m guessing that for many readers the answer is “not in the last 5 years”. Once common, the high street travel agent is a prominent casualty of the wave of e-commerce that swept over economies in the early 21st century.</p><p>Another way of thinking about the fate of the high street travel agent is that the travel industry underwent a paradigm shift (or a “disruption” in the ghastly parlance of silicon valley). In other words the <em>dominant methodology</em> we used to solve the problem of “how do I book travel?” shifted from high street travel agent to online travel agent in the space of a few years, rendering the former largely obsolete.</p><p>These kinds of disruptions are increasingly common given the rate of technological advancement and evolving consumer preferences. For companies and other large organizations, navigating and creating these disruptions/paradigm shifts is the business of strategy.</p><h3>Strategy as paradigm</h3><p>Strategy is about determining the framework that an organization will use to solve the problems they’ve decided to be interested in, or in other words the paradigm the organization will base itself around. This might sound strange, but looking back the most successful business strategies have been paradigm shifts: the car, the home computer, the iPhone, e-commerce, cloud computing, and so on. New ways of solving old problems (transport, communication, commerce, information storage) that came to dominate.</p><p>So how does data, and data science, fit into all this? Recall the earlier emphasis on the creative nature of paradigm shifts: experiments and data collected in one paradigm can’t indicate or point to what a new paradigm will look like. Put another way, data is <em>paradigm dependent</em>, i.e. the data collected to understand and optimize paradigm A is largely irrelevant to paradigm B.</p><p>For example, let’s go back to the high street travel agent. Let us say that these travel agents were data savvy and extensively optimized their business using AI and analytics, picking the right streets to set up shop, opening times, staffing levels etc. All of that optimization and data would have been… entirely useless in the face of the rapid shift online of the travel industry. Similarly, no amount of analysis on data collected on high street travel agents in the 80s would have hinted at the e-commerce paradigm shift bearing down on them.</p><p>For this reason strategy cannot be led by, or emerge from, data. This is not an exhortation for strategy teams to ignore data (they hardly need my encouragement there…), but rather not to fall into the trap of assuming the key to great strategy is in poring through reams of statistics and models. Use data, but not so much that it distracts from the real job of strategy: creatively reimagining how to solve problems in ways that will be better than what’s gone before.</p><h3>Data as executor</h3><p>So what is data good for? Paradigms come into the world as imperfect ideas that sketch out an approach for solving problems, but leave countless details and rough edges for scientists to pour over and iron out. This process of refinement: creating hypotheses, collecting data, carrying out experiments and so on, is what most scientific practice consists of. Hence Kuhn’s term for it: “normal science”, work designed to understand and optimize the dominant paradigm.</p><p>The role of data science and analytics within businesses and similar organizations is no different: to execute, understand, and optimize the chosen strategy. This is what most data work in large companies actually consists of: running tests, optimizing processes, giving insight into how well strategy is being executed, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Dw2pgtrcqNnDI3BP" /><figcaption>Paradigm refinement workshop (Photo by <a href="https://unsplash.com/@sieuwert?utm_source=medium&amp;utm_medium=referral">Sieuwert Otterloo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>A possible objection that might be forming in the reader’s mind is that data can often show the flaws in strategy, leading it to be overturned in favour of another. Whilst this is entirely correct, it rather shows the paradigm cycle in action: problems arise with a paradigm through the process of normal science, leading to crisis and a new paradigm being selected. This process may happen much faster in, say, a business than in physics laboratories, but the fundamental mechanics at play are the same.</p><h3>Strategy driven data</h3><p>To summarize, strategy cannot be “data driven”. Rather, the strategy that is employed dictates the kind of data that is collected and the nature of the AI and analytics that are carried out. Even between companies competing on similar products, different strategies will generate different sets of data, and AI that function and optimize in different ways.</p><p>Sure, a strategy will probably involve <em>using data</em>, but in the 21st century that’s pretty much a given. Data is the new oil, in the sense that it’s ubiquitously used by almost everyone for almost everything. The question is not whether you’re going to use data, but<em> </em>what exactly you’re going to do with it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99ca9f2bd18e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/your-strategy-should-drive-your-data-not-the-other-way-round-99ca9f2bd18e">Your strategy should drive your data, not the other way round</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evaluating Machine Learning Models]]></title>
            <link>https://medium.com/data-science/evaluating-machine-learning-models-f5c5635db0d7?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5c5635db0d7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[decision-making]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Thu, 24 Sep 2020 14:23:11 GMT</pubDate>
            <atom:updated>2020-09-25T14:52:41.391Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/getting-started">Getting Started</a></h4><h4>How to understand whether an ML model is really doing what you want it to</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CvdmdXEY95lUE-5Y" /><figcaption>Photo by <a href="https://unsplash.com/@flpschi?utm_source=medium&amp;utm_medium=referral">Feliphe Schiarolli</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When the hard work of building a model is done, someone has to make the final decision of whether to press “go” and release the model into the wild. This is the crucial moment for a data scientist, when they find out whether they’re going to see their work take flight, or whether it’s back to the drawing board. It’s also an important process for the decision maker, as wrongly approving an algorithm can result in disruption and cost if crucial mistakes are only caught after the fact, or worse don’t get caught at all.</p><p>Understanding the key factors in making that decision is therefore important both for the data scientist developing algorithms, and for the people who are going to be relying on them. In this blog we’ll talk through some key components of this decision, using a model built for the purpose on open source <a href="https://www.imdb.com/interfaces/">IMDB data</a>.</p><h3>Setting the scene</h3><p>Context is king when understanding a model, so let’s spend some time sketching that out. Let’s say our model is being built on the request of a movie studio executive disappointed with a string of movies with lack-lustre reviews. The idea behind the model is that it will be used to predict the review score of a movie when it is being pitched, and so give the movie commissioning team more information on whether to approve or reject the pitch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b2ZIPpw80GJmHBNd" /><figcaption>Photo by <a href="https://unsplash.com/@jakobowens1?utm_source=medium&amp;utm_medium=referral">Jakob Owens</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We’ll use the IMDB user ratings as a guide for quality for our model. This is a rating assigned by users of the site, with anyone being able to leave a rating (let’s leave aside the question of whether that’s actually a good metric for quality for now!). After a couple of weeks of crunching the numbers, our data scientist has a model to present to the movie executive, using a random forest regression model. Is it good enough to be used by the team? Read on….</p><h3>Understanding “Accuracy”</h3><p>The most natural question to ask is how accurate is the model. This seemingly simple question can be approached in a number of ways, and it’s easy to throw a barrage of accuracy metrics out there but still leave stakeholders none the wiser. Here are some tips on presenting this in a way that will help both you and your stakeholders get a grip on what your model is really capable of:</p><ol><li>Pick the right accuracy metric(s)</li><li>Visualize the error metric(s)</li><li>Choose a relevant accuracy benchmark</li><li>Develop custom metric(s) that speak directly to the model context</li><li>Investigate relevant sub-populations within the dataset</li></ol><h4>1. Pick the right accuracy metric(s)</h4><p>Picking the right metric is a question of understanding how the model will be used. In our case:</p><ul><li>Will all movie pitches be ranked by predicted score, with the top X being approved? If so, we should explore accuracy metrics that specifically evaluate ranking accuracy</li><li>Will all movies above a certain predicted score be approved? When we’re dealing with a more binary yes/no approval threshold, we may want to think of this as a classification problem with metics like Cohen’s Kappa or F1</li><li>Is it more important to get some movies “right” than others, for example based on budget size? Here we’d need to weight the model to prioritize certain movies, and the accuracy metrics we report also</li><li>Will this be used to help understand the predicted review score of both individual movies and a variety of movie portfolios being pitched? In which case we are most interested in the point estimate accuracy of each movie.</li></ul><p>Let’s say for now that the last option is closest to how the model will be used. For this reason we’ll pick R2 as that gives a good feel for how close an individual prediction will be to the true outcome.</p><h4>2. Visualize the error metric(s)</h4><p>Now we’ve selected our error metric we can confidently tell our stakeholder that the R2 for our model is 0.3 (on the validation set of course). It’s hard to say from merely quoting this what’s going on though, so visualizing the metric can go a long way to giving stakeholders a more tactile sense of accuracy. For R2 the most sensible visualization is probably a scatter plot, which you can see below.</p><p><em>Model Predicted rating vs. Actual Rating on validation set (2020 movies)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*aw9-BUAHaGRF8RK6" /><figcaption><em>Model Predicted rating vs. Actual Rating on validation set (2020 movies). “numVotes” indicates the number of votes that comprise the average rating, giving a feel for how the accuracy varies for movies with fewer votes</em></figcaption></figure><p>This tells us a lot more than just a simple R2 metric. Firstly we can see that though there’s definitely a relationship between the prediction and the actual result, there’s a lot of uncertainty around any given prediction.</p><p>Also, we see that there’s a “bunching” of the predictions around the average 5–7 ratings which we see less of in the actual results. This reflects a lack of predictive power in the model: it doesn’t have enough information to differentiate high from low so it “guesses” results around the average score as that will give it the minimum R2. This also means that it will generally understate the difference in average review rating between portfolios of movies, as it will drag both a portfolio of good movies and bad movies towards the mean value.</p><h4>3. Choose a relevant accuracy benchmark</h4><p>The best way benchmark your algorithm is to compare it to how decisions would be made if your algorithm wasn’t used. In our case, instead of rolling out a complex model to our movie commissioning team, it may be easier to come up with a simple heuristic, say: “the movie rating will be similar to the average of the rating of the last five movies made by that director”. This works as it’s easy to understand and can be quickly assessed by anyone with an internet connection. So how does our algorithm do?</p><p>In this case, the heuristic gives an R2 of 0.14, so is about half as accurate as our model. That’s good news, and gives us some confidence that our model is adding value over more simple approaches.</p><h4>4. Develop custom metrics that speak directly to the model context</h4><p>Standard model accuracy metrics like R2 and RMSE can be quite abstract, so it can often be valuable to create metrics that are more tactile and closely related to how the model will be used. In this case, a good metric is the average actual rating of the top 100 movies with the highest predicted rating, which is 8.92. This is better, as it gives a clear real-world view of what the model will achieve.</p><p>Again, we can compare this with our benchmark heuristic, which gives a similar value of 8.13, so our algorithm is 0.79 better by this metric. Is this enough to justify the algorithm roll out? That can depend, but now our movie commissioner can make a trade off of “if I spend the necessary time/energy to get my movie commissioning team to adopt this, I’ll get on average 0.79 higher ratings for my movies”.</p><h4>5. Investigate relevant sub-populations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*3MWXf531DEUaHDPnLydSbw.png" /><figcaption>Photo by <a href="https://www.pexels.com/@caleboquendo?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Caleb Oquendo</a> from <a href="https://www.pexels.com/photo/woman-standing-in-front-of-video-camera-2918590/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><p>In many cases it’s necessary to assess not just overall accuracy, but accuracy for different groups or sub-sections of the data. In our case we may want to check the model is not treating male or female directors differently, a not uncommon problem that our movie commissioner is keen to avoid. Looking at the R2 we see 0.29 for female directors and 0.3 for male directors, so it seems the model is pretty much equally accurate for these populations, which is good news. Of course that’s not the whole story on bias, and we’ll touch on this more later.</p><p>Now we have a good handle on how accurate the model is, let’s get to know how it “thinks” a little better.</p><h3>How the model makes decisions</h3><p>It’s common for people to switch off at this stage, especially given the somewhat nerdy image of data science and the much-publicized “black box” nature of many algorithms. Despite the complexity though, a lot can be done to give a non-technical audience a feel for how a model is making decisions.</p><p>Firstly, there’s a feature importance plot. Most ML packages make a form of this available, which indicates how much each variable is being used by the model to make decisions (in the case of a random forest, how much the feature decreases impurity when splitting on that feature).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Xb7qK33izGKGPwB" /><figcaption>Model uses 70+ features, so limited to top 20 features only</figcaption></figure><p>Here we see some expected behavior, and some weirdness. Many of the important features relate to how previous movies by the director, writer or actors have rated, which makes sense. However, the length of the movie appears to be the most important factor in the model, which doesn’t sound right. Let’s use a partial dependence plot to understand what’s going on:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*q_LCjVn5zon2ttbj" /></figure><p>Here we see what the issue might be: there appears to be two “levels” of review score, one for “shorts” i.e. 10–30 minute mini-movies, and one for more normal length features. It seems that, on average, shorts get higher reviews than full length features. This may well be a real phenomenon with shorts either attracting a different reviewer audience, people liking shorter films, or some unseen feature that is more prevalent in the “short” movie category. On the face of it though, film length doesn’t intuitively <em>feel </em>like it should impact quality, so we may decide to remove this feature provided it doesn’t heavily impact accuracy, a decision that can be discussed with our movie commissioner.</p><p>We can also check some of the other high impact features to see if the model is treating them in a sensible way. Let’s take the next most important variables, the mean rating of the director’s last 5 movies, and the mean of the actor’s last 5 movies (weighted by how highly ranked in the movie billing the actor is).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*G6bdokbKlfew0JzKhe8YSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*8Bj_Q8Gf6poR0Wm1EqL8dg.png" /></figure><p>Both of these give fairly sensible and intuitive results, the higher rating the actors and directors have had in the past, the more likely future movies by them will get a high rating. Looking closer does reveal some interesting behavior however: there’s an S shape to the variable impact, implying that beyond a certain level good previous ratings don’t matter. With the director’s most recent movies this S curve decrease in impact mostly happens where there’s little data, but for the mean actor rating we see that quite a lot of the movies occur after the variable stops having a significant impact, implying beyond around 6.5–7 it doesn’t significantly influence average rating. Bad actors can tank a movie, but great actors can’t make a mediocre movie great.</p><p>In summary then, our model is not that dissimilar to the heuristic we calculated earlier, only it takes into account a wider range of similar heuristics, takes the length of the movie into account (unless we remove this feature). Of course, one could go into more detail here, but this gives a good overview for our movie commissioner.</p><h3>Showing examples in the flesh</h3><p>For all the clever visualizations and interpretation techniques, there can often be little substitute for just showing the model in action on some examples to give people a feel of how it’s working.</p><p>One should take care in picking examples, it can be very tempting to go “oh that’s a weird result, let’s hide that”, but that’s a bad habit. Firstly one would be misleading the stakeholder on how the model will actually perform, but also exposing oneself to later criticism: if you show some of the bad results up front and the model still gets approved, the stakeholder can’t say they weren’t warned. However, if you’ve hidden inconvenient results the stakeholder may reasonably ask why they’re seeing odd outputs only after agreeing to deploy the model.</p><p>Whilst there should always be an element of randomness in picking examples, using a purely blind random selection may not be the best approach if one is only going to show a few, as the odds of them being unrepresentative are quite high. Instead, there are more guided approaches worth discussing.</p><p>Firstly, one can pick well known examples that the stakeholder is likely to be familiar with, like below where I’ve picked randomly from the top 50 most popular (by vote count) movies of 2020. This has the advantage of being something the stakeholder can engage with and have an opinion about, and so mean you’re more likely to get meaningful feedback and nuggets of context you wouldn’t otherwise have got.</p><p>Alternatively, if there aren’t obvious examples, one can pick the most representative in certain categories, for example the “typical” action movie or romance, in the case of films. If the categories aren’t obvious, a clustering algorithm can be used to identify sub-populations in the data to sample from. This has the advantage of allowing one to identify the centroid and so enabling you to pick the most mathematically representative example from a given cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zz2YVH7XljZaypYI" /><figcaption><em>Randomly selected from top 30 movies on IMDB in 2020 by number of votes. Note all movies in 2020 were held out as the validation set for the model</em></figcaption></figure><p>Now we’ve got our examples, we see some good predictions (“Da 5 Bloods”) and some not so good ones (“Extraction”). Whilst one can’t draw general conclusions from these, they can make the general conclusions shown in previous charts more real.</p><p>Note that I’ve put the heuristic we discussed earlier in as well, so one can see how predictive that is. This is useful as we see that though the model outperforms the heuristic in aggregate, it won’t always outperform it (e.g. “Gretel &amp; Hansel”, “Sonic”, and “Extraction”). Whilst statisticians have had it drummed into them that an average performance improvement is not a guaranteed performance improvement, a few examples can help clearly make the point to those who haven’t been… er… lucky?… enough to have that training.</p><h3>How the model will impact the world around it</h3><p>One of the main things on a stakeholder’s mind is in what way the model will change how their teams, systems or processes function. When a model optimizes it usually doesn’t do so in a neutral way. Rather, models aim to improve a specific outcome in part by favoring some groups and disfavoring others. Therefore, it’s important to understand and communicate to the stakeholder what the impacts might be.</p><p>For example, we saw in the feature importance plot that the model paid particular attention to horror movies. Does this mean they could be disfavored by the algorithm? This could be a key point for our movie commissioner to understand, as this may mean that in using this model less horror movies get made. Let’s look at that in more detail along with the gender of the director and the short/full movie distinction that we also touched on earlier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/0*uiu32JzMdi4QwGPW" /></figure><p>Here we see that the model doesn’t penalize female directors, though these remain under-represented in the data. Also horror movies do better rather than worse out of our model, so again little concern there.</p><p>On “shorts” however we see a much more pronounced impact, the model has identified that shorts generally get higher scores than full length movies, and so is really pushing those kinds of movies even beyond what we see in the actual results. This may not be what the movie commissioner wants, and so would be a key consideration in deploying the model. One option might be to have separate thresholds or quotas for shorts and regular movies, all the way through to forcing a similar distribution of scores for shorts and full length movies in the output.</p><p>One can do this with any set of metrics or features. The skill here is in determining which output characteristics are relevant to the model context, and what the stakeholders are likely to be interested in.</p><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hxedCfgG7jsEC5vs" /><figcaption>Decisions Decisions (Photo by <a href="https://unsplash.com/@aj40?utm_source=medium&amp;utm_medium=referral">TAHA AJMI</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>Hopefully you now have a pretty good understanding of how the model is functioning, and whether you’d want to use it in the wild. There’s no right answer here, it’s a question of how valuable the gains are and how those trade off against the complexity and consequences of rolling out such a model (this author would lean to the heuristic on the basis that it’s simpler and doesn’t perform that much worse).</p><p>The overarching point here though isn’t about movies or this particular model, but in the steps and approaches one can use to explain a model to a non-technical stakeholder, and also the right questions to ask as a stakeholder when an ML model is presented to you. To recap:</p><ol><li>Pick the right error metrics, visualize them, and don’t be afraid to get creative in how you measure accuracy</li><li>Use techniques like variable importance and partial dependence plots to get a feel for how the model “thinks”</li><li>Use examples, and be careful &amp; fair in how you pick these</li><li>Think through and illustrate how the model will impact the system it is deployed into</li></ol><p>As AI is integrated into more and more of our lives and professional environments, it’s vital for people to learn the right questions to ask of these algorithms to ensure they’re actually adding value and not causing unintended harms under the surface. Great data science often uses black box algorithms, but it should never be a black box.</p><h4>Further Reading</h4><p>If you’re interested in exploring these ideas further I’d highly recommend a few of the papers and blogs below, as they’re a sample of the literature the author has drawn inspiration from in writing this blog.</p><p>Model cards (Timnit Gebru et. al.) <a href="https://arxiv.org/abs/1810.03993">https://arxiv.org/abs/1810.03993</a> and <a href="https://modelcards.withgoogle.com/about">https://modelcards.withgoogle.com/about</a></p><p>Cassie Kozyrkov on why AI makes mistakes: <a href="https://towardsdatascience.com/dont-trust-ai-10a7df520925">https://towardsdatascience.com/dont-trust-ai-10a7df520925</a></p><p>A great discussion on AI explainability on the TWIML AI blog: <a href="https://open.spotify.com/episode/2n7PVTEcoQMish70AYoHY4?si=k6PwmW7USSOkx2HtYjQVQg">https://open.spotify.com/episode/2n7PVTEcoQMish70AYoHY4?si=k6PwmW7USSOkx2HtYjQVQg</a></p><p>A comprehensive guide to interpretable Machine learning (Christoph Molnar) <a href="https://christophm.github.io/interpretable-ml-book/">https://christophm.github.io/interpretable-ml-book/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5c5635db0d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/evaluating-machine-learning-models-f5c5635db0d7">Evaluating Machine Learning Models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working out the kinks in your data science setup]]></title>
            <link>https://medium.com/data-science/working-out-the-kinks-in-your-data-science-setup-3cc82e02f31d?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cc82e02f31d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Sat, 20 Jun 2020 14:13:20 GMT</pubDate>
            <atom:updated>2020-06-20T14:13:20.213Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide to some common issues that data science teams face</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*woJ6s7eALHNqfqZn" /><figcaption>Photo by <a href="https://unsplash.com/@jeshoots?utm_source=medium&amp;utm_medium=referral">JESHOOTS.COM</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>This is a companion piece to this blog on </em><a href="https://towardsdatascience.com/the-key-elements-of-effective-data-science-teams-468676b16440"><em>data science team setup</em></a></p><p>As many practicing data scientists will tell you, the path to effectively integrating data science into an organization can be anything but smooth. There’s a lot of hype and expectation around what data science can deliver, but these hopes can falter when the day-to-day reality of dealing with data problems, bugs, and computation limits sinks in. This blog is a guide to some of the common problems a data scientist, or data science team, can encounter with some suggestions on how to go about dealing with them.</p><h3>I spend all day finding and managing data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oet6SQR7FQiDVuOt" /><figcaption>The data you need is in here somewhere… (Photo by <a href="https://unsplash.com/@randomlies?utm_source=medium&amp;utm_medium=referral">Ashim D’Silva</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>This is one of the most common issues, and is a sign that the data scientist(s) have been hired without enough thought as to the environment and setup they need to be successful. Managing data is a constant challenge for most organizations and the ability to reliably ensure the right data is in the right place at the right time is crucial for effective data science. Therefore, to fix this we usually need to be looking at our <strong>data engineering setup</strong>. Are there enough of them, and do they have the time to focus on your needs?</p><p>A commonly quoted ratio of data engineers to data scientists is 1:2, but really it’s a function of how messy the data you need is. If everything is already well organized into easily query-able and reliable data stores then you may need only 1 data engineer for a few data scientists. If your data is spread all over the place in assorted prem and cloud stores, with a few excel spreadsheets and SQL server instances mixed in for good measure, you may need more of a 1:1 ratio.</p><h3>I built this awesome model but the stakeholders hated it</h3><p>This can be one of the most frustrating problems, as weeks or even months of work go up in smoke when an influential stakeholder turns round and says: “nope”. This tends to happen when crucial points or requirements get missed in the scoping process, or got lost when the algorithm is being built out.</p><p>Looking closely at your <strong>product management processes</strong> is therefore key to understanding why these issues happen and preventing them recurring. I say product management <em>processes </em>as whilst it can be that product management dropped the ball, it may also be that data science or engineering teams aren’t adhering to the product requirements, or aren’t listening enough to the stakeholders. It can also happen that stakeholders invent new requirements towards the end of a project, or forget to mention crucial points at project initiation.</p><p>A couple of ways of avoiding these issues are:</p><ul><li>Ensure algorithm requirements are clearly documented at the start of the project, and that all stakeholders have reviewed and signed off on that documentation</li><li>Where possible, get a first version of the algorithm output in front of the main stakeholders as quickly as possible. This doesn’t have to be fancy, or using the same model and feature set that the final algorithm will use. The aim is to show “this is roughly what the output will look like”. Discussions centered around a first draft tend to be much more productive than discussing an algorithm in the abstract.</li></ul><h3>I spend all day chasing bugs!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q7Jm3eW4SfO0EIyV" /><figcaption>Photo by <a href="https://unsplash.com/@miteneva?utm_source=medium&amp;utm_medium=referral">Maria Teneva</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Bugs, unfortunately, are a fact of life and whilst I’d love to be able to give a “magic bullet” style tip to avoid them, I’m afraid I’ll have to leave you disappointed. However, bugs shouldn’t be <em>all </em>a data scientist deals with and if you find that you’re slowing down on delivery as you’re always bug-hunting then it may be time to take a look at the <strong>engineering setup</strong>.</p><p>Deploying ML algorithms can be done in a number of ways, from the data scientist’s building the entire pipeline themselves, using off the shelf or in-house AI platforms, to the engineering teams re-writing a data scientist’s specification directly into the software itself. The nature of bugs can also vary hugely from sporadic weirdness to full blown system failures. That said, some common themes around buggy data science are still worth discussing.</p><h4>Data scientists writing production code without the necessary training/skill</h4><p>Now this isn’t to say that data scientists <em>shouldn’t</em> write production code, many can and do just fine. But whilst data science training does involve a lot of coding, it tends not to focus on creating great software engineers. So if your algorithm is constantly acting out, it may need someone with more relevant training and experience to re-write it.</p><h4>Algorithms being poorly suited to the engineering realities</h4><p>You can build the best product recommendation algorithm in the world but if it takes 20 seconds to render on the page then customers will abandon the site in droves. Ensuring an algorithm works well in the context it will operate in, and not just in the training environment, is crucial and the reason the role of Machine Learning Engineer has risen to prominence over the last few years. Fixes here can be as simple as removing features that are reliant on buggy data stores, or as complex as re-visiting the algorithm type and training/scoring approach.</p><h3>I spend all day answering questions</h3><p>Algorithms and data products generate questions. How well is it working? How does it behave in certain circumstances? Should we be exploring different features or datasets? These can come from both stakeholders but also team members as they seek to grow and improve the algorithm/product. Here the skills of an <strong>analytics </strong>are best employed: rapid querying, data visualization, and storytelling.</p><p>A common problem can arise when an algorithm is attracting a number of pertinent questions, but there’s too little (or no) analytics resource on hand to answer them. This tends to push the analytics work onto the other team members such as product, data science and engineering which slows down delivery (and usually isn’t their core expertise). This can lead to the paradoxical situation where a known issue doesn’t get fixed because the team are too busy answering questions about it!</p><h3>A lot of iterations are needed to get things working properly</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-y0FPlumuQtTLBf2" /><figcaption>Photo by <a href="https://unsplash.com/@steve_j?utm_source=medium&amp;utm_medium=referral">Steve Johnson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Any algorithm or product requires a few iterations to work out the kinks and really hit its stride, but this should be a question of refining a broadly good solution, rather than continually getting things way off and having to regularly go back to the drawing board. Customers and stakeholders only have so much patience.</p><p>It can be tempting to blame these kinds of issue solely on product management, and though that can be part of the problem, merely focusing there can miss a broader issue. Whilst product management are indeed in charge of picking the right tools and approach to solve a given problem, for that solution to really work the data scientists and engineers who build it also need to have detailed knowledge of the problem and context, and thoroughly understand why a particular solution has been chosen.</p><p>Things go wrong when there’s too much of a gap between the data scientist/engineer, and the business or problem area they’re building for. There can a few causes here:</p><ul><li>Heavy rotation of people on/off a project</li><li>Processes that don’t allow data scientists and engineers to participate in, and contribute to, the solution design and scoping process</li><li>Data scientists and engineers not having the time needed to get “under the skin” of the customers or stakeholders they’re building for.</li></ul><p>None of these are conceptually difficult to fix, but making the changes in processes and setups can take time and persuasion. It will be worth it though, great products and algorithms are built by people who are experienced and knowledgeable about what they’re working on. Conversely data scientists and engineers who just “do what they’re told” are a recipe for disengagement and mediocre software.</p><h3>Summary</h3><p>When deploying machine learning into an organization it can be a constant challenge to ensure you’re always in a position to work effectively, and aren’t side-tracked by tasks that aren’t part of a key data scientist’s skill set. I’ve sketched out how one can think about some of these issues, but the actual solutions will be dependent on the context of the organization you’re working in. In essence, whenever you’re persistently hitting issues that prevent you from getting the job done, don’t just bash your head against them but step back and look at the broader set up, and where you may need more help or a different approach. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cc82e02f31d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/working-out-the-kinks-in-your-data-science-setup-3cc82e02f31d">Working out the kinks in your data science setup</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The key elements of effective data science teams]]></title>
            <link>https://medium.com/data-science/the-key-elements-of-effective-data-science-teams-468676b16440?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/468676b16440</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 21:32:38 GMT</pubDate>
            <atom:updated>2020-06-06T21:32:38.454Z</atom:updated>
            <content:encoded><![CDATA[<h4>What to think about when building a data science team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ra9B3IHRfVLf6bUN" /><figcaption>Photo by <a href="https://unsplash.com/@marvelous?utm_source=medium&amp;utm_medium=referral">Marvin Meyer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s a lot of blogs, articles, and literature on data science techniques, but much less on how one actually goes about setting up a data science team. What kind of skills are needed? How should they fit together? Who should do what? In this blog I’ll sketch out the crucial elements of a data science setup, introduce a cast of characters, and give an overview of how they work together.</p><p>In this context, I’ll be talking about data science teams that build algorithms into software, and so have to worry about code performance, stability, operations etc. This doesn’t apply to data scientists who do analytics or statistics, where the algorithms may be just as complex but the outcome is more a decision, report, or dashboard.</p><h3>Take this seriously, but not literally</h3><p>Terms such as “product manager”, “engineer”, and “data scientist” don’t have consistently defined meanings, so if you’re reading this and find yourself going “wait, that’s not a data scientist’s job!” then please, bear with me. This is less about the <em>job title</em> of the person who needs to, say, write unit tests to ensure the code is stable, and more about pointing out that <em>someone </em>needs to do this, and whilst that someone is doing that, they aren’t able to do other things like model development or feature exploration.</p><p>Also, I’m not implying any kind of organisational design. There’s a number of ways to organize teams on data science initiatives, from the squad model championed by Spotify to the more traditional vertical where all of these roles would report up into a single overall lead. The “right” setup will be a function of the organizational culture, resources, maturity and local constraints, and whilst I do have some opinions here I’ll save those for another time.</p><h3>Before we get started…</h3><h4>The “Product”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hqG9y6LqDqlTufjl" /><figcaption>Photo by <a href="https://unsplash.com/@petekasprzyk?utm_source=medium&amp;utm_medium=referral">Peter Kasprzyk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here I’ll refer to the piece of software we’re building algorithms for as the “product”. Products are rarely driven by a single algorithm, rather being built out of a number of algorithms, business rules, user interfaces, APIs etc. For example, let’s say we’re building a tool to optimize routing taxis around a city. This will likely be made of a number of elements:</p><ul><li>Matching algorithm to select which customers to link to which drivers</li><li>Routing algorithms to both get the customer to their destination, and the driver to their customer</li><li>Location recommendation algorithms to indicate to drivers when and where customers are likely to be</li><li>Pricing algorithms to set an acceptable price for both customer and driver</li><li>An API to serve all of the above in real time to an app or desktop device</li></ul><p>Now, not all of these have to be in the same product. My point is rather that most data science driven products involve multiple algorithms and components working together, so one has to think more broadly than optimizing a single algorithm or function.</p><h4>Stakeholders</h4><p>Before we get to the team proper, there’s one vital group of people that we need to talk about: stakeholders. They come in a range of guises: customers, executive leadership, other internal teams, even other data scientists. In essence, a stakeholder is any group that either you want to influence, or wants to influence you.</p><p>Now you may be thinking: that’s simple! It’s whoever we’re building the algorithm for! But that would be a naive mistake. Any large scale data science project has a range of stakeholders, and properly understanding who they are, what they want, and how much influence they have (or should have) is crucial to the success of any data science, or other, product.</p><p>For example, back to our taxi routing product, who are your stakeholders? Well the obvious two are:</p><ul><li>the <strong>customers </strong>who want to quickly and cheaply get from A to B, and who you want as many of as possible using your product</li><li>The taxi <strong>drivers</strong> who will want as many customers as possible, without too much driving, and who again you want to attract to using the product</li></ul><p>But is that it? Well, likely not. Let’s say you check your email on Monday after releasing the first version of your routing product and you see a few more stakeholders coming into view:</p><ul><li>The head of <strong>marketing </strong>wants you to ensure waiting times are below 2 minutes in evenings, as that’s what they’ve just committed to in an expensive and high profile advertising campaign</li><li>The <strong>finance </strong>team wants to understand how you minimizes fuel expenditure, to keep costs and carbon emissions no higher than they need to be</li><li>The <strong>CEO </strong>wants to know why your product left them standing in the rain for 30 mins last night waiting for a taxi (yes, this sort of thing does happen!)</li></ul><p>Welcome to your network of stakeholders! Not one of these people/groups can be entirely ignored, but likely they can’t all get everything they want either. So how to trade off all these stakeholders against each other? Well that leads us nicely to our first team member….</p><h3>Meet the team</h3><h4>Product management</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iPcR5LG_hDDkJ1A9" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>They are the team quarterback, or captain if you prefer English football to the American kind. They’re in charge of making sure the team is working on the right things at the right time, and managing the complex web of stakeholder needs.</p><p>Firstly they act as a <strong>triage </strong>point. Businesses are often fast moving environments with large numbers of vocal stakeholders: exposing the team directly to that can be hugely distracting. Product management does the vital job of working out which stakeholder asks need to be worked on immediately, which can wait, and who should do the work. This latter part is especially important with Machine Learning/AI as they’re much hyped technologies that everyone wants to be using. In my experience at least half the asks for an AI solution will be better solved by more prosaic approaches such as improving user interfaces, analytics, or data engineering.</p><p>Next they take the broad, unstructured requests from stakeholders and <strong>scope </strong>them into discrete tasks for the team to digest and work on. Stakeholders don’t know the inner workings of the algorithms &amp; code, so can only point to aspects of the product that they don’t like or feel should be added. Translating these broad asks into a specific alteration that make sense, works in tandem with all the other moving parts, and will fully answer the need articulated is the nuanced art of scoping. This is a difficult but vital role and doing it well is crucial to the health of the product. In my experience, most project failures are scoping failures.</p><p>Finally, great product managers aren’t just sausage factories for taking in a blend of stakeholder requests and churning out prioritized, scoped work, but have their own <strong>vision </strong>of how the product should evolve. You may roll your eyes at the hackneyed term “vision”, but the ability to think ahead and say “this is what this product should become” in a way that actually makes people want to get out of bed and make it happen is a crucial element to success. Without vision, products tend to turn into a Frankenstein’s monster of what different stakeholder groups want the product to be, doing a bunch of them acceptably but none of them excellently. A good test is that if you cleared all the requests from stakeholders away, would you still have a clear view of what you want to do?</p><h4>Data science</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sMztJxbNHHnagj5q" /><figcaption>Photo by <a href="https://unsplash.com/@cheaousa?utm_source=medium&amp;utm_medium=referral">Ousa Chea</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You’re probably surprised that I’ve taken nearly 1,000 words to get to the data scientists. Whilst I should probably apologize for that I won’t, as it illustrates how much careful thinking needs to be done before one starts writing code.</p><p>Their first responsibility is selecting the right <strong>algorithm objective</strong>. For example, should our taxi routing aim to maximize the number of trips, the number of customers, the total revenue of a given taxi, or something else? Are some customers or journeys more important than others (e.g. female customers at night)? Though the objective can often seem simple at first, for any problem there’s a large number of possible objectives and careful selection of the right one is a key part of the data scientist role.</p><p>Once an objective is defined, that naturally leads to how the data scientist will <strong>assess accuracy</strong> against that objective. Again, this can take a number of forms, our taxi data scientist can choose the average error in trip length prediction, either in absolute terms (which will focus the algorithm on longer journeys) or as a percentage of actual trip time (which will focus the algorithm on shorter journeys). Also, it’s important to pre-declare what the right accuracy criteria are in advance of starting, to avoid the less-than-scrupulous approach of just generating a large number of statistics and picking the ones that look best.</p><p>For any algorithm to give good results the <strong>training data</strong> must be carefully curated. Algorithms are trained to reflect the patterns and relationships present in the data we feed them, so poorly thought through training data can cause real problems for a data science product. For example, if our routing training data-set is only generated from private vehicles, and we’re designing an algorithm for taxis which can use bus lanes, we may set the right objective and get great accuracy on the hold out sample, but poor results in the field.</p><p>Lastly, the <strong>algorithm approach</strong>, which includes model selection, design and feature engineering. These are the mainstay of data science and where often most of the time is spent. There’s already a lot of literature on this topic, so I won’t add to that here apart from calling out a point that often gets missed: it’s important to think through how <em>stable </em>you need your model outputs to be. For example, if our routing model gives a slightly different route each time you ask it to get you from A to B that probably isn’t an issue, but if our pricing algorithm gives a different price every time you ask the same question that’s going to cause you a lot of problems!</p><p>Once we’ve an algorithm ready it’s time to get it in front of some users, and for that we’ll need…</p><h4>Engineering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bj0ccnFKeZSFuJ9t" /><figcaption>Photo by <a href="https://unsplash.com/@murilosilvapr?utm_source=medium&amp;utm_medium=referral">Murilo Silva</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A truly great data science product can only be built on the foundation of great engineering. Your math can be perfect, but if the model is down half the time or full of bugs then it won’t be much use. Engineering are therefore crucial players, and have a number of roles.</p><p>In the ideal world, the data needed to train and build models would already exist, in a nice clean format with all the relevant attributes stored in a single, easily accessible data-set. Sadly, data nirvana is always out of reach as even in the best data environments the needs from data are constantly evolving as the business and contexts evolve. Therefore, you’re going to need engineering help in<strong> data creation</strong>. This can involve a fair mix of detective work, ETL creation and especially for large datasets, creativity around cost effective storage and retrieval solutions.</p><p>Once you have your data and model you’ll need to string it together into a piece of software that can execute reliably when needed, or a “<strong>data pipeline</strong>” to use the jargon. This can be as simple a weekly “batch” run to drop a small text file in a shared folder, to massive real-time “streaming” pipelines involving huge amounts of complex data. Also, if the algorithmic output is being consumed by another service (e.g. an app) then further engineering may be needed to ensure the data pipeline meets the standards of the service consuming it.</p><p>Finally, once this is all built there’s <strong>operations</strong>. Is the system up and running when it needs to be? Are all the downstream components and datasets correct and up to date? Do the outputs make sense? Keeping the ship running in a way people can rely on it may not be the sexiest part of data science, but you’re nowhere without it. Now, any of my engineering colleagues reading this will be raising their eyebrows at the notion that good operations are just their job, and they’re right: it’s everyone’s job. But engineering is often the first line of defense in analyzing and triaging bugs, and building resilience into the system.</p><h4>Analytics</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yjfw0apsnUoDC1qS" /><figcaption>Photo by <a href="https://unsplash.com/@srd844?utm_source=medium&amp;utm_medium=referral">Stephen Dawson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Finally, any large scale data science powered product generates questions: is the product meeting its objectives? How do we know the new version of the algorithm is better than the old one? How well are the algorithms responding in specific scenarios? Answering these questions is the job of analytics.</p><p>Firstly, any data science driven products are built around specific <strong>targets</strong>, these are usually linked to the algorithm objectives, but can often be broader. For example our taxi routing product might have targets such as a certain number of customers or revenue. Tracking how the product is doing against that target, and explaining over or under performance, is a key analytical role.</p><p>Next up, any product can generate an almost limitless number of <strong>questions </strong>from stakeholder groups. How fuel efficient is our taxi routing? How do waiting times vary by city? Are we over or under-serving different customer profiles? Whilst some of these will require the data scientist to dig into the guts of the algorithm, in most cases intimate algorithm knowledge isn’t necessary and you’re better off utilizing the rapid querying &amp; data visualization skills of an analyst.</p><p>No product starts out great, to really beat and stay ahead of the competition <strong>testing </strong>is crucial. Websites and apps often have the luxury of randomized AB testing, but even without that creative analysts with good statistical knowledge can find ways to design tests to confirm that an amazing idea or new algorithm feature really is everything it’s hoped to be.</p><h3>Bringing it all together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gA1hQOoPBwfOPmQu" /><figcaption>Photo by <a href="https://unsplash.com/@joshcala?utm_source=medium&amp;utm_medium=referral">Josh Calabrese</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve presented the team in a sequential order from stakeholder &gt; product manager &gt; data scientist &gt; engineer &gt; analyst, and whilst there’s some truth to that in reality all team members need to be involved in all parts of the project life-cycle to some degree. Product managers can’t craft the right solution without understanding the data science, data scientists need to build algorithms that will actually run given the engineering realities, and engineering needs to keep a close eye on the product objectives when building out infrastructure.</p><p>Most importantly, a culture of shared success and mutual cooperation is vital to making the team effective. Victories need to be shared by all team members, and problems something that the entire team needs to think through to overcome. If bumps in the road are met with finger pointing then you’ll quickly find that problems stay hidden and decisions are taken to protect egos, not drive results. Ultimately, customers and stakeholders aren’t going to care who exactly screwed up or deployed this or that feature: they just see a product that either does or doesn’t do what they want.</p><p>Building a great algorithm is hard, but building a great data science product can be downright fiendish. The outline I’ve provided should give you a view on the key pieces that need to be in place for you to have a fighting chance. A good team setup won’t guarantee your success, but a bad one will guarantee your failure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=468676b16440" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-key-elements-of-effective-data-science-teams-468676b16440">The key elements of effective data science teams</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Joy of Simulation]]></title>
            <link>https://medium.com/data-science/the-joy-of-simulation-9906f1d5c604?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9906f1d5c604</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[glengarry-glen-ross]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[monte-carlo]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Wed, 20 May 2020 15:44:31 GMT</pubDate>
            <atom:updated>2020-05-20T15:44:31.899Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Monte Carlo simulation to set test parameters and select algorithmic approaches</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H-lCl6ZxEJSLNKGw" /><figcaption>Statistician’s paradise (Photo by <a href="https://unsplash.com/@johnschno?utm_source=medium&amp;utm_medium=referral">John Schnobrich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>One of the trickiest problems in data science, among a crowded field, is picking the right approach and tools for a task which doesn’t have a “ground truth” to compare different options against. By “ground truth” I mean examples of the real outcome that you’re trying to predict or evaluate.</p><p>So for example, if you’re wondering whether ARIMA or Prophet is better at forecasting ice cream sales, you can take a date in the past where you sold, say, 10 ice creams and if ARIMA predicts 4 and Prophet predicts 11, you know Prophet is better as it got closest to reality i.e. the “ground truth”. Simple! OK, it usually isn’t <em>that </em>easy, but the principle is simple…</p><p>But what if you’re trying to work out whether temperature or day of week has the highest impact on ice cream sales? Or how many extra ice creams you sold because of all the “it’s hot, maybe buy some ice cream?” posters you put around town last week (look, advertising isn’t my strength…)? Here you have no “ground truth” as you don’t know what the impact of temperature and day of week <em>really </em>are, or how effective your posters <em>really </em>were. All you know is that different approaches and packages spit out different estimates of these values, but telling which of these estimates are good and which are less reliable is tricky, as you’ve nothing to compare them (against apart from each other).</p><p>So what to do? Well in these cases a great tool is the humble Monte Carlo Simulation. So named after the Monte Carlo Casino in Monaco in the 1940s, these simulations essentially consist of building a virtual version of whatever system you’re interested in, which you then run various scenarios through a large number of times to generate a “brute force” solution to a problem.</p><p>Because building these simulations is usually straightforward, often achievable with only a few lines of code, as soon as the probability mathematics get tough it’s usually easier to just fall back on a Monte Carlo simulation. For anyone who is regularly dealing with complex probability problems (i.e. any data scientist/analyst), this makes them an invaluable tool.</p><p>How are they useful for the “no ground truth” problem? Using a Monte Carlo simulation you can create a rough copy of the situation you’re dealing with, and then <em>create your own ground truths </em>for comparing different approaches against.</p><h3>Always Be Closing</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxT9IgnBxp2Jp8WWtVK%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fmovie-glengarry-glen-ross-xT9IgnBxp2Jp8WWtVK&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2FxT9IgnBxp2Jp8WWtVK%2Fgiphy.gif%3Fcid%3D790b76115a0708ed9b249d6b5b36b24e958b30b5acbeae02%26rid%3Dgiphy.gif%26ct%3Dg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="183" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd79fe9c62e79ab4f24847470121946a/href">https://medium.com/media/bd79fe9c62e79ab4f24847470121946a/href</a></iframe><p>Let’s illustrate with an example: say you’re a statistician tasked with developing a testing strategy for sales scripts for use by sales agents to sell real estate. The information you have is:</p><ul><li>Sales agents make on average 10 sales a day, normally distributed with a standard deviation of 4</li><li>Your boss wants to test around 50 ideas this year. Given the constraints around testing, each test will be a 50:50 split with 50 agents in each bucket</li><li>The uplift of any new script is evenly distributed between +10% and -15% with respect to the previous script (i.e. any given new script has a 40% chance of having a positive impact)</li></ul><p>You have four test options:</p><ol><li>A one tailed T test with a confidence level of 95%</li><li>A one tailed T test with a confidence level of 99%</li><li>A one tailed T test with a confidence level of 60%</li><li>Roll out every new script!</li></ol><p>Now, one way to solve this is to get out a pencil and paper and do a whole bunch of algebra to work out which strategy is optimal, and if that’s what floats your boat: be my guest! But if you’re anything like me you’ll probably find it easier to build a Monte Carlo simulation of this scenario and then let it do all the hard work for you.</p><p>Let’s do that, running the above scenario, say, 500 times to see what outcomes we get with each test strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/0*9X7Puz80ZHbUUV69" /></figure><p>That’s perhaps not the result one might have expected. What’s going on? To answer that let’s take a look at a single run of the simulation, and see how the cumulative impact of the tests evolve over time. As you can see, the loosest test criteria (confidence at 60%) wins because though it makes a fair few bad calls, it selects so many more winners that the final net effect is overall much more positive than the other, more stringent, test settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*mY2razC0-AfDmJlV" /></figure><p>The crucial point is that in this scenario we’re not trying to optimize the result of a <em>single </em>test, but rather we’re looking to maximize the number of correctly identified winners minus the number of incorrectly identified losers in a <em>large portfolio of tests</em>. More generally, if one is doing a high volume of testing one can often afford to be much less confident in the results of any one given test. This also has the happy consequence of usually meaning one can get away with less data per test, enabling either less time testing or more tests in the time available, but I digress.</p><p>Now, you may be objecting that the folk creating advertising scripts (or in a usual data science setup: product managers/leadership) are rarely candid or able enough to say things like “funnily enough, we actually get this wrong 60% of the time”, so perhaps things aren’t as clear cut as in my example?</p><p>The beauty of a Monte Carlo simulation however is that one has tremendous flexibility to see how different scenarios play out. Perhaps the sales <a href="https://www.youtube.com/watch?v=mikCQoDG2Ls">leads are weak</a> and you want to see how that’s going to influence your test? Maybe <a href="https://www.youtube.com/watch?v=Lbp3ozjl-qM">third prize is you’re fired</a> so you want to go for a safer strategy that maximizes the probability of getting a result above a certain threshold, regardless of expected value? Or as above, you’ve a sneaking suspicion the success rate of the new sales scripts just won’t be that high? All of these scenarios can be plugged into the simulation to see how they might pan out, usually with only small changes to the code.</p><h3>Building your Casino</h3><p>When building these simulations you can generate the data yourself, as I did above, creating sets of random numbers that match the distribution of the data you’re interested in*. You can usually build a half decent simulation with less than a hundred lines of code, though of course you can make these as complex as you like. Do beware highly complex simulations however, as these can lose some of the simple interpret-ability that makes this an attractive tool.</p><p>An alternate approach is to take the actual data from the system you’re interested in, and make modifications to it in line with what you’re interested in. So in the above example one would take the actual sales history and adjust sales up or down in line with the simulated test impacts. The advantage here is that you can be much more confident that the simulation matches reality, the disadvantage is you’ve a lot less control if you want to see how changes to distributions in the data affect the outcome.</p><p>* Another quick digression: don’t just assume all your data is normally distributed as this can bite you pretty hard when it’s not true! For example most sales distributions are not normal, with a small number of salespeople/ products/ ads driving a large share of the sales.</p><h3>A data scientist’s best friend</h3><p>In case you weren’t already convinced, another reason to like this approach is that when speaking with stakeholders, simulations are often more intuitive. One can talk more easily to the assumptions being made, why certain outcomes/runs end up the way they did, and how changes to the input assumptions affect the outcome. I tend to find stakeholders are often more receptive to this than “I did a boatload of complex probability math and you better hope I didn’t screw it up”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ddeBBlw_Rr6W3akM" /><figcaption>I got this right, promise (Photo by <a href="https://unsplash.com/@roman_lazygeek?utm_source=medium&amp;utm_medium=referral">Roman Mager</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>So if you’re not sure which causal impact/uplift package is right for your particular problem and data distribution? Need to check if your multivariate regression is correctly sizing coefficients? Not sure how quickly your algorithm will respond to a changing environment? Can’t work out if a statistical test is going to work with your data distribution? Give your old buddy Monte a call.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9906f1d5c604" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-joy-of-simulation-9906f1d5c604">The Joy of Simulation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasts in the time of Corona]]></title>
            <link>https://medium.com/data-science/forecasts-in-the-time-of-corona-5541bc1df8f7?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5541bc1df8f7</guid>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Wed, 06 May 2020 21:13:33 GMT</pubDate>
            <atom:updated>2020-05-06T21:13:33.168Z</atom:updated>
            <content:encoded><![CDATA[<h4>Now more than ever we need to be cautious in how we create and use forecasts</h4><p>Working in travel as a data scientist during this crisis, I’m frequently asked how data can be used to forecast when the travel industry will return to some sort of health. My answers all boil down to: “buddy, right now I can barely give you an accurate forecast of what’s going to happen <a href="https://www.bbc.com/news/world-us-canada-52407177"><em>this afternoon</em></a>”. But that got me thinking: how should we be approaching forecasting at the moment? And what even is forecasting anyway?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*10nAsEjVn5poQhzB" /><figcaption>The Tardis: the ultimate forecasting machine (Photo by <a href="https://unsplash.com/@casman?utm_source=medium&amp;utm_medium=referral">Charlie Seaman</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><h3>What is forecasting anyway?</h3><p>The underlying problem with forecasting is that we can’t actually predict the future. When we say we’re going to “forecast” something that we’re actually saying is that we’re going to do one of three things (or a combination):</p><h4>Time series projection</h4><p>Here, we focus on the history of the metric we’re trying to forecast, and assume the patterns in that history are going to persist into the future. So if we’re forecasting ice-cream sales we’ll only look at data on previous ice-cream sales, and assume the peaks in summer and weekends will continue.</p><p>Once we’ve made that assumption, all we need to do is build models that accurately describe that history (or “time series”), and then use those to project forward into the future. Models such as ARIMA fit into this category.</p><h4>Modelling</h4><p>Here, one builds a model of the metric one is interested in, using data on the various factors that have the most influence on it, or correlation with it. Our ice-cream forecast, in this case, would involve understanding the relationship between ice-cream sales and temperature, rainfall, week day, school holidays etc. A key point here is you need to both be confident you’ve captured all the key factors that influence the forecast metric, and have a good forecast of those factors themselves. So if you have a bad weather forecast, that’s going to hurt your ability to build a good ice-cream sales forecast off it.</p><p>Weather forecasts and predictions of sporting events and elections are good examples of this kind of forecasting.</p><h4>Scenario analysis</h4><p>Here we make no attempt to divine trends or relationships from data, and instead just imagine what could happen and then see how that would play out in the various systems/processes we’re interested in. You may be thinking “hang on, that’s just guessing?”, and yes, that’s exactly what it is. But that’s also the appeal: you’re not lulling people into a false sense of security with seemingly-precise numbers, and the human imagination can do a better job than any machine in speculating how once-in-a-lifetime scenarios might play out.</p><p>This is a simple approach, but more common than you might think. Pension, investment, and mortgage providers do a lot of this, and a bank’s stress test is essentially a pessimistic scenario analysis. Obviously these tend to be more useful before everything has gone sideways but can be useful in the midst of a crisis when one isn’t sure how things are going to play out.</p><h3>Why is forecasting so hard?</h3><p>For forecasting to work you need to be confident in two things:</p><ol><li>You have a good handle on the major factors that influence the outcome i.e. low <strong>Aleatoric</strong> uncertainty in <a href="https://en.wikipedia.org/wiki/Uncertainty_quantification#Aleatoric_and_epistemic_uncertainty">statistical jargon</a>.</li><li>There aren’t any major unpredictable events that will occur out of nowhere and mess things up i.e. low <strong>Epistemic</strong> uncertainty</li></ol><p>The more eagle-eyed of you may have spotted an issue with the second point: if said major events are unpredictable, how do we know there aren’t enough of them to sway the outcome? Let’s explore that point a bit more with an example.</p><p>Imagine a forecaster who works for the UK government, and is tasked by the tourism minister with forecasting how many international visitors will arrive in the UK by month next year. Looking at the <a href="https://www.ons.gov.uk/peoplepopulationandcommunity/leisureandtourism/datasets/monthlyoverseastravelandtourismreferencetables">data</a>, they think to themselves: “well this looks pretty regular and stable, this shouldn’t be too hard”, and build a forecast like the one below using a seasonal ARIMA model. They include some nice, clear 95% prediction intervals so that the tourism minister can get a feel for how much uncertainty there is around this forecast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/0*5hccXogkScEMuHER" /><figcaption>A really bad forecast of UK visitor numbers</figcaption></figure><p>How did they do? Whilst I don’t have the updated numbers for visitor counts to the UK, at the time of writing air tickets are down to almost zero, according to <a href="https://www2.arccorp.com/">industry data</a> helpfully summarized by BCG <a href="https://public.tableau.com/profile/the.boston.consulting.group#!/vizhome/BCGsTravelRecoveryInsightsPortal/PortalHome?publish=yes">here</a>, so we can assume total visitors are doing something similar. In summary, not great.</p><p>What happened? Well, one could argue that nothing really went wrong: there’s a 95% lower bound prediction interval and we exceeded it, but we should expect that 1 time in 20, so, what’s the fuss? The problem with this reasoning is that we didn’t just exceed the lower bound a bit. We crashed through it and didn’t stop until we hit the ground. Using the normal distribution assumptions underlying the prediction intervals, this kind of event has a probability of 0.000002% i.e. we should only observe something like this once every 5 million years!</p><p>Rather, the problem is that the prediction intervals are only a statement about <em>aleatoric </em>uncertainty, based on what the algorithm has observed in the data it was trained on. Put another way, forecast prediction intervals are themselves <em>only a forecast </em>of the uncertainty, and so only show the minimum bounds of real-world uncertainty.</p><p>In fairness, epistemic uncertainty is un-quantifiable (if it was quantifiable, it would be aleatoric) and so hard to show on a chart. One thing the forecaster could have done would have been to accompany the forecast with a view on how the crises of 2001 and 2008 affected travel numbers. This would have made it clear that they were just offering a “normal year” forecast and in the real world, anything can happen. Hindsight is a wonderful thing.</p><h3>Forecasts are usually wrong, plan accordingly</h3><p>As we’ve seen, whatever forecast you happen to be looking at is going to be a bit wrong most of the time, and a lot wrong some of the time. Because of this, it’s vital to always know how much error you are OK with, and what you’ll do when errors exceed those bounds.</p><p>The most important part of any forecaster’s job, therefore, is to understand what kind of errors the client or system one is forecasting for is most sensitive to. If you’re forecasting for inventory, will it be more expensive to over-stock or run out? If you’re forecasting to set a target, will it be better if targets are routinely beaten or frequently missed? A common mistake forecasters can make is to sink hours into different forecasting techniques and models, but only minutes working to understand how the inevitable errors will impact the system or client they’re building a forecast for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lr-FJ2t18vMvC2-J" /><figcaption>A good forecaster doesn’t just minimize error, they select the right kind of error to minimize (Photo by <a href="https://unsplash.com/@victormuunoz?utm_source=medium&amp;utm_medium=referral">victor muñoz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>Luckily, this kind of thinking comes naturally to us when the decision is framed properly. I’ll bet the last time you threw a party you bought more food and drink than you ended up needing. Far from being irrational, this is because the cost of over-forecasting is low (excess can be stored cheaply) and the cost of under-forecasting is high (is there anything sadder than a party that’s run out of booze?).</p><h3>Adapt, don’t predict</h3><p>As you’ve probably gathered, data-based forecasting isn’t going to be all that helpful over the coming months, as we have large amounts of both aleatoric and epistemic uncertainty. We don’t have a good handle of how, or whether, people will want to travel in the months ahead. Even when we do start to understand that, any forecast built off that understanding runs a high risk of getting shredded by unpredictable events, such as a second wave, political decisions, economic impacts (can’t fly if there are no airlines), or, on a happier note, new treatments or vaccines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pzMljpE36VJJTmo3" /><figcaption>You’ll have to wait, old friend (Photo by <a href="https://unsplash.com/@tim_front?utm_source=medium&amp;utm_medium=referral">Timur M</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>So what can we do if we can’t forecast? The most effective approaches are going to be a combination of:</p><p>Using <strong>Scenario analysis</strong> to outline several possible outcomes, and mapping how they impact what you’re interested in. Note that this doesn’t have to be fancy, and in many industries and fields there are many of these already being developed that can be used off-the-shelf. In doing this, it can be tempting to pick a “most likely“ scenario, based on the information you have, and then design plans around that. But that’s falling back into the forecast trap: the information you have now can be irrelevant tomorrow.</p><p>Clearly assess the <strong>sensitivities to risk</strong> of the client, system, or decision(s) you’re making, and compare those sensitivities to the possible scenarios. Note that you should be trying to make decisions that will maximize your ability to meet your objectives <em>regardless </em>of which scenario plays out, even if it’s not the <em>optimal</em> one in each particular scenario. Your risk tolerance also comes into play here; you may be prepared to accept a decision that runs the risk of ruin in some scenarios if there’s a big pay-off in others.</p><p>Finally, given circumstances are likely to continue to evolve quickly, exploring how you can improve your ability to be <strong>flexible and responsive</strong> will enable you more effectively to meet that change, and less reliant on the guesswork inherent in scenario analysis.</p><p>So when would be the right time to return to forecasting? Well, if possible you may not want to. Forecasting is an uncertain business, and the less you can base your decision making on it, the more robust your decision making will be. That doesn’t mean you should never fire up an ARIMA model or look at a weather forecast again, many decisions require <em>some </em>view of what is going to happen in future, but it does mean that you should always have in the back of your mind: what would I do if this turns out to be really wrong? How can I quickly adapt if it does?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*anA67H_w4RZCQbkS" /><figcaption>Do you feel lucky? (Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>)</figcaption></figure><p>In an important sense, forecasting is now more than ever much like picking winners on the stock market: there’s a huge degree of randomness, few can do it well, and it’s very hard to distinguish the talented from the merely lucky. But, we can use the same tools and approach that good fund managers use to manage their portfolios: understanding their risk tolerance, picking strategies like diversification that can weather a range of outcomes, and ensuring they can flexibly move funds around to take advantage of opportunities. And of course, never forgetting the age-old investing maxim: Past performance is no guarantee of future results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5541bc1df8f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/forecasts-in-the-time-of-corona-5541bc1df8f7">Forecasts in the time of Corona</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[There are no facts, only interpretations]]></title>
            <link>https://medium.com/data-science/there-are-no-facts-only-interpretations-abc70a48014b?source=rss-3c9383cf6bc6------2</link>
            <guid isPermaLink="false">https://medium.com/p/abc70a48014b</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[coronavirus]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Daniel Hill]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 18:38:39 GMT</pubDate>
            <atom:updated>2020-04-28T09:58:54.972Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Coronavirus crisis is illustrating the importance of knowing your data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aK_Vz_eS_J9cKqeTtsLK_A.jpeg" /><figcaption>Friedrich Nietzsche, data expert (image copyright Adobe)</figcaption></figure><p>“There are no facts, only interpretations”, according to Nietzsche anyway. That may seem odd: data, with its apparent precision, structure, and clarity, may seem the very definition of objective certainty, but that’s an illusion. The reality is that data is always a subjective and imperfect expression of how the people who collected it view the world.</p><p>The Coronavirus outbreak has proved an excellent case in point. On the one hand it’s an extremely transparent crisis. Case and death counts are published daily, with open-source data sets for anyone to analyze and model to their heart’s content. And yet, despite this, getting good answers to even basic questions is exceedingly tough.</p><p>What I’ll do in this post is outline some of the questions I use when thinking about data, and then apply them to an open-source Coronavirus dataset to see how they affect our ability to answer a couple of common questions: how dangerous is the virus, and how well are governments managing the crisis?</p><h3>Questions to ask about data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-O0fCo8jWvA57ym_" /><figcaption>If you don’t police your data, no-one else will. Photo by <a href="https://unsplash.com/@mattpopovich?utm_source=medium&amp;utm_medium=referral">Matt Popovich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Why was the data collected?</h4><p>When collecting data, a host of decisions have to be made about what to collect, what to miss out, and how to interpret various circumstances. Understanding <em>why</em> data was collected is crucial in helping you understand the decisions that resulted in the data in front of you and should govern then the way you think about that data. A data set that built by someone who’s trying to sell you something should be thought of differently to one built by a more neutral party. A data set that feeds a crucial process or piece of infrastructure is different from one that has been sat on a server for a year or two without anyone looking at it.</p><p>Also, without knowing the why, you’ll always be asking yourself “wait, why did they do that?!” as you uncover various gremlins and oddities in the data.</p><h4>How complete is the data?</h4><p>There’s no such thing as a truly “complete” dataset, as the process of data creation necessarily involves deciding not to collect information on certain entities or attributes of those entities. There are always more things one could measure, so thinking through what you’re <em>not </em>seeing in the data, as well as what you are seeing is crucial.</p><p>Often, you’ll be dealing with sampled data, as collecting data on an entire population is either unfeasible or too expensive. In those cases, it’s crucial to understand how and why the data is sampled in the way that it is, as well as the consistency of the sampling approach that was chosen. Often, the sampling approach chosen for use A makes it invalid for uses B, C and D, as we’ll see later on when considering Coronavirus mortality.</p><h4>How accurate is the data collection process?</h4><p>Data accuracy can be influenced by anything from test accuracy, the stability of the system generating the data, or the knowledge and motivation of manual data collectors. Do bear in mind that there’s no guarantee that just because data is all in the same table, it’s all equally accurate; some may be well maintained and precise, and others a hot mess.</p><h4>How can human error or bias creep in?</h4><p>Sometimes, the people collecting the data are knowledgeable, careful, well-resourced, and have plenty of time to do their work well. Other times that’s…. not so much the case. Similarly, where there are strong incentives to see certain outcomes it will be hard for many to completely resist the urge to nudge the data in that direction. Then there’s always good old fashioned fraud…</p><h4>Has the data been collected consistently?</h4><p>None of the above points are necessarily consistent over time, location or data collectors. New leaders or stakeholders of the data collection teams may mean changes in data collection policy. Evolving technology or broader context may alter the approach or scale of the data collection. It can also be just as simple as a bug that’s found 6 months in where the fix can’t be back-dated.</p><h3>Coronavirus data</h3><p>The <a href="https://ourworldindata.org/coronavirus">Our World in Data</a> site has a lot of excellent charts that are sourced daily from the <a href="https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide">European Centre for Disease Prevention and Control</a>. There are other sites, but they all have similar sources: a collection of reported cases and deaths by country.</p><h4>Why was the data collected?</h4><p>In this case the data is primarily collected by health authorities of each country to inform the relevant national government of how the virus is spreading, and the strain that’s placing on their health services.</p><p>This means that the data tends to focus on cases that impact the health system, and is more tailored to the needs of each national government with less of a view to global consistency. That’s not to say there aren’t efforts to align practices, but the needs of the national government will come first.</p><h4>How complete is the data?</h4><p>As has been well recorded, test availability is limited in many countries so the data is far from representative. As a side note, this point is crucial when identifying where countries are in the infection curve: have they hit peak or just maxed out their testing capability?</p><p>Tests are usually not carried out at random. Rather only cases presenting symptoms (and sometimes only very severe symptoms) get tested. This makes any inference about mortality pretty suspect.</p><p>Similarly on death counts, different countries are counting different things as Coronavirus deaths. Some have only been counting in-hospital deaths, others including care-home and other non-hospital deaths which can have a very large impact on totals.</p><h4>How accurate is the data collection process?</h4><p>No test for a virus is completely accurate, and Coronavirus tests have false positive and negative rates like any other. There are a number of tests available and some appear to be <a href="https://www.theguardian.com/world/2020/apr/19/scientists-raise-concerns-about-quality-of-uk-covid-19-tests">much less accurate than others</a>, so again we need to be careful in making comparisons between populations tested using different tests.</p><p>Also note the date grain in this data is by date of report, which can give an odd weekday/weekend cycle as not all sources report on weekends (no the virus is not taking a break on weekends).</p><h4>How can human error or bias creep in?</h4><p>The pandemic is a large, frightening, global crisis where often already stretched health infrastructures are adapting to huge problems quickly. Errors are to be expected.</p><p>Some countries may not be entirely transparent in their reporting practices either. Accusations of downplaying the numbers have been leveled at <a href="https://www.theguardian.com/world/2020/apr/17/china-denies-cover-up-as-wuhan-coronavirus-deaths-revised-up-50">China</a> and <a href="https://www.businessinsider.com/coronavirus-russia-doctors-say-government-is-covering-up-cases-2020-3">Russia</a> for example.</p><h4>Has the data been collected consistently?</h4><p>Data collection is not consistent by country (or state): different testing strategies, different tests, and different ways of measuring deaths all mean that we’re usually not comparing apples to apples. That’s not to say that metrics are never comparable between countries that have made an effort to keep their metrics aligned, but we should assume that’s not the case until shown otherwise.</p><p>We also expect metrics within a country to change over time, as more (and better) tests become available and governments adapt their metrics to better reflect their understanding of the crisis (e.g. including care home deaths in the totals reported).</p><p>In summary, this is a pretty sketchy set of data and we’ll need to be very careful in drawing conclusions from it. Let’s look at a couple of questions to see the impact this has on our ability to draw conclusions.</p><h3>How dangerous is Coronavirus?</h3><p>Before we dive in, it’s worth noting that the infection &amp; mortality rate of the virus are not the same as your personal risk from the pandemic. If you’re lucky enough to avoid the virus but then get into a car accident where the ambulance that normally takes 5 minutes to arrive takes 50 minutes you may well become a victim of the pandemic, if not the virus. These “collateral” deaths are not trivial, with <a href="https://www.economist.com/graphic-detail/2020/04/03/covid-19s-death-toll-appears-higher-than-official-figures-suggest">early analyses</a> indicating they may double or more the reported toll in many countries (though some of that will be unreported viral deaths as mentioned above).</p><p>Also, this question implicitly assumes there’s one strain of the virus with one mortality rate, when there may be <a href="https://www.scmp.com/news/china/science/article/3080771/coronavirus-mutations-affect-deadliness-strains-chinese-study">multiple strains with varying mortality rates</a>.</p><p>Still, let’s focus on the mortality of the virus itself. Taking a quick look at the <em>World in Data</em> report for the 21st of April, we see 169,859 deaths and 2.43m cases;this gives us a 7.0% mortality rate.</p><p>As we already noted, though, things aren’t that simple. Both case and death counts are being heavily skewed by how governments are administering tests, which tests they’re using, and how they’re recording deaths. Untangling the impact of these distortions for the global dataset will be extremely tricky, if not impossible.</p><p>Of course, we will answer this question in time and studies are already underway (with, thankfully, <a href="https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30243-7/fulltext">much lower mortality than 7%</a>). My point is more that, even with a great deal of daily-updated global data at our fingertips, we’re still getting answers that are likely an order of magnitude off. Instead, for this particular question we need to craft a bespoke data-set, one with randomized sampling and consistent tracking.</p><p>Let’s try another question and see if we fare any better.</p><h3>How well are governments managing the crisis?</h3><p>As mentioned above the focus is generally on deaths directly from Coronavirus, so if some anti-virus measures are inadvertently causing deaths elsewhere it’ll be some time before we know it. Let’s put that concern to one side for now, however, and again focus on virus mortality. Also, there’s some debate on whether we should be comparing absolute numbers or deaths/cases per million. Given that I’m interested in comparing Switzerland (population 9m) to the US (population 328m) I’ll look on a per-capita basis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/0*Aay9hEx8yOQhG4rR" /><figcaption>Data from: <a href="https://ourworldindata.org/coronavirus">https://ourworldindata.org/coronavirus</a> (snapshot taken on 19th April 2020)</figcaption></figure><p>Spain is clearly having a tough time, as has been widely reported. The UK, France, and Italy have virtually identical curves. The US hasn’t quite hit the gradient of Spain, Italy or the UK yet. Switzerland looks like Germany but worse, and South Korea remains the positive outlier.</p><p>So what can we conclude about the differing strategies and competencies between these countries? Well apart from telling everyone to be more like South Korea (be well-prepared, far in advance) it’s hard to make confident judgments about the effectiveness of their measures, and there’s no obvious pattern between lock down ‘strength’ and death count. As mentioned earlier, we don’t know how different data collection practices are exacerbating or hiding differences between these countries.</p><p>Also, to properly answer this question we need a great deal of additional data on how much factors like population density, virus variants, <a href="https://edition.cnn.com/2020/03/16/opinions/south-korea-italy-coronavirus-survivability-sepkowitz/index.html">demographics</a>, smoking prevalence, climate and so on are shaping spread and mortality, versus those factors that are attributable to government policy, such as testing, face masks, lock-down, and social distancing. Only once we’ve controlled for these factors will we be able to point to specific decisions and policies and see what kind of impact they really had.</p><p>The point here is not to let incompetent governments off the hook; some governments definitely seem asleep at the wheel. Rather, the point is that despite the wealth of data at our fingertips, we find ourselves needing more context and different data to properly answer this question.</p><h3>All the gear, no idea</h3><p>We’re increasingly blessed with huge amounts of data to play with, and cool tools to use in doing so. Whilst that’s a great thing, it definitely makes it easier to fall into the worst habit of analysis and data science: jumping into data we don’t really understand and quickly making a bunch of cool charts &amp; metrics to show everyone.</p><p>Though not particularly sexy, spending the hours (yes, hours) to really understand the data you’re working with, and how it relates to the problem you’re trying to solve, is the difference between great analysis and, well, shiny-looking garbage. You’ll be surprised how often you find the data you started working with isn’t actually what you need at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DbiestNCYunXOvIc" /><figcaption>Data Science is like cooking, the magic is in the ingredients. Photo by <a href="https://unsplash.com/@videmusart?utm_source=medium&amp;utm_medium=referral">Syd Wachs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The question set outlined in this piece is, I hope, helpful in going through that process. Undoubtedly, they’re not exhaustive, and you may have your own you like to use. The bottom line is this: treat data not as objective truth, but more like a reflection of what whoever created it thought was important and feasible to log at the time. Put another way, analytics and data science are like cooking, you can make almost anything bubble away in a pot, but if you haven’t chosen the ingredients properly you may not want to eat the result.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abc70a48014b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/there-are-no-facts-only-interpretations-abc70a48014b">There are no facts, only interpretations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>