<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dan Frankowski on Medium]]></title>
        <description><![CDATA[Stories by Dan Frankowski on Medium]]></description>
        <link>https://medium.com/@dfrankow?source=rss-abc2df62e05b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*R3sTj0t0EaWvFkJa3WOITw.jpeg</url>
            <title>Stories by Dan Frankowski on Medium</title>
            <link>https://medium.com/@dfrankow?source=rss-abc2df62e05b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:14:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dfrankow/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to become a data scientist, part 3: tell people about your work]]></title>
            <link>https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-3-tell-people-about-your-work-87f4edf1d896?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/87f4edf1d896</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[communication]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Thu, 14 Apr 2022 15:53:56 GMT</pubDate>
            <atom:updated>2023-03-14T19:41:21.726Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-1-find-a-good-problem-2971442227cc">part 1</a>, I suggested you find a good problem. In <a href="https://towardsdatascience.com/how-to-become-a-data-scientist-part-2-try-to-solve-the-problem-f9b9746f6eb9">part 2</a>, I suggested you try to solve it. Here I discuss how to tell people what you did.</p><p>Let me start with a story. I was presenting some work, a couple weeks of digging in data to define a segmentation of search queries that people could use to think about search. I looked out at the audience, saw most people waiting, checking their email. I said, “Let’s get started” and several laptop lids closed. I said, “I’d prefer it if you closed your laptops.” The rest closed, but a couple of people were still staring at their phones. I called one out, saying, “Hey A, something interesting?”, pressing to get those phones down. After a short tussle, mostly everyone was paying attention, and I started.</p><p>Why did I work so hard? Why offend people? Because I’m old? Sure, I remember life before smartphones. I like to get my cane out and shake it at the young ‘uns until they get off my lawn. But also because <em>it doesn’t matter what I say, only what they hear</em>. <strong>The purpose of data science is its effect on others.</strong> Readers, listeners, decision-makers, users.</p><p>Below, I describe two situations: you 1) did an analysis, or 2) built a product (or feature or tool). Maybe you did both.</p><h3>The analysis goal: tell a story, with evidence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*MINPiHVA7A1thCa13LaUAQ.gif" /><figcaption>You need a good question and some progress .. or to tell people what they already wanted to hear. <a href="https://dilbert.com/strip/2012-06-23">dilbert.com/strip/2012–06–23</a></figcaption></figure><p>There is some debate over whether you should engage in “story-telling.” Some consider it over-hyped. It’s not. People remember and repeat stories.</p><p>They also remember and repeat memes and pop songs. You can use those to get your point across, but it’s harder and often doesn’t fit exactly. We’ve all seen people trying to get a point across like <a href="https://twitter.com/ResearchMark/status/849628943187664896/photo/1">Research Wahlberg</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*oZ7dNZydINuGW2OfGSMg-A.jpeg" /></figure><p>It’s fun, but people often remember the joke, not the point.</p><p>By contrast, stories can and should be tailored very specifically to your point, like the cranky old man story I opened with.</p><p>Ideally, you answered some part of the high-level (“research”) question you asked when you <a href="https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-1-find-a-good-problem-2971442227cc">chose the problem</a>. Now:</p><ul><li><strong>Describe the question and the answers.</strong> I’ve seen plenty of writeups without a clear question (why did they do this?), or without a clear answer (what happened, precisely?).</li><li><strong>Highlight a few takeaways.</strong> People are only going to remember a few things, say three or fewer. Spoon feed them!</li><li><strong>Make recommendations.</strong> Sometimes I have observations without recommendations, because I want to get something out. Readers hate this. They want <a href="https://en.wikipedia.org/wiki/Persuasive_writing">persuasive writing</a>. That means you have a position, and you’re trying to convince someone of it, ideally with evidence. Honestly, it’s also easy to convince without evidence, but to live with myself, I prefer evidence. If you chose an actionable question, these recommendations should come easily.</li><li><strong>Be brief, but don’t tease.</strong> Analyses often have an executive summary. Research papers usually have an abstract. No one has time to read everything. Your summary should be self-contained, with the main results, and some numbers. Ideally, it is intriguing enough that people want to read more. Don’t tease. Some abstracts state the question and methods precisely, but leaves the answer for the full writeup. Those authors suck. Gimme the answer. If it’s interesting, I’ll want to know more anyway.</li><li><strong>Be clear and compelling.</strong> Write well. Present well. There are many books on this. I should do another post with English writing recommendations, and another on making a presentation. If it’s a live presentation, watch and respond to your audience. Keep them with you.</li><li><strong>Show multiple lines of evidence.</strong> If you have numbers, that’s good. If you have user feedback, that’s also good. If you’ve looked at edge cases or important use cases, that’s also good. More different sources can make a case much stronger. Numbers can lie, as can user stories. It’s harder to make everything lie.</li><li>Finally: <strong>don’t just do what people asked, do what’s right</strong>. People often don’t know what they want, or want the wrong thing. Part of your value is figuring that out. (“Wrong” might be “unhelpful” or “unethical” or “nonsensical”, it varies.) This is even more true as you get more senior. People expect you to figure things out (“handle ambiguity”). That’s what they pay for.</li></ul><p>You don’t have to fully solve your problem or answer your question. You just need a good question and some progress: more insight than before you started. You may have only worked a couple of weeks, especially if you want to check in to see if you’re headed in the right direction.</p><p>Most people prefer fast to complete. See <a href="https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-1-find-a-good-problem-2971442227cc">part 1</a>, wherein Jeff Bezos said, <em>“Most decisions should probably be made with somewhere around 70% of the information you wish you had.” </em>Academics jokingly aim at an LPU, a “<a href="https://en.wikipedia.org/wiki/Least_publishable_unit">least publishable unit</a>”. A ten-page academic paper can be months of work instead of weeks, but the point is everyone breaks up their work.</p><h3>The product goal: tell a story, with evidence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*UKIHA2AHtB9WPG-KrfwSZg.png" /><figcaption>Even if you built something, you still have to describe it. <a href="https://xkcd.com/1838/">xkcd.com/1838</a>.</figcaption></figure><p>Yep, same goal.</p><p>Say you built a recommender. Is it any good? With no proof, a reputable business isn’t going to drop a tool on its users, nor will an academic conference publish a result.</p><p>So the story is often, “Here is why I built the product (or feature or tool), and evidence that it works.” For a product at a large Internet business, this may be an <a href="https://en.wikipedia.org/wiki/A/B_testing">A/B experiment</a> showing user experiences in aggregate. For an internal tool, you can do a user study (like <a href="https://fredhohman.com/papers/19-gamut-chi.pdf">this</a>).</p><p>There is a burgeoning field of explaining <a href="https://medium.com/fiddlerlabs/case-study-explaining-credit-modeling-predictions-with-shap-2a7b3f86ec12">machine learning</a> <a href="https://towardsdatascience.com/a-gentle-introduction-to-algorithmic-fairness-46c825bdaf5d">models</a>, <a href="https://towardsdatascience.com/humans-choose-ai-does-not-6512d49f7c46">that</a> <a href="https://towardsdatascience.com/https-towardsdatascience-com-that-debugging-predictions-using-explanations-861cc94cb875">I’ve</a> <a href="https://towardsdatascience.com/causality-in-model-explanations-and-in-the-real-world-9f57e17a9138">written</a> <a href="https://towardsdatascience.com/should-you-explain-your-predictions-with-shap-or-ig-9cabe218b5cc">about</a>. The short version is: people should not believe results from a “black-box” model (one whose assumptions you cannot examine). You should know more about why a model is producing its predictions, to prevent machines from <a href="https://towardsdatascience.com/a-gentle-introduction-to-algorithmic-fairness-46c825bdaf5d">picking up and amplifying the wrong thing</a>.</p><p>There is a related mistake I’ve seen professionally: when people define a goal metric (an objective, numeric way of measuring success), and then treat any positive movement in a number as success. <a href="https://towardsdatascience.com/humans-choose-ai-does-not-6512d49f7c46">No metric encompasses everything you want.</a> If your goal metric is user engagement, you may get clickbait. Closely related is <a href="https://en.wikipedia.org/wiki/Goodhart%27s_law">Goodhart’s law</a>: <em>“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes,”</em> often restated, <em>“When a measure becomes a target, it ceases to be a good measure.”</em></p><p>You should always look at both quantitative measures (towards the goal) and qualitative measures (examples, edge cases), to keep yourself honest. That would force you to stare at the clickbait and say, “Is this really what I want?”</p><p>If you often find that your numbers look good, but looking at individual cases doesn’t, consider coming up with “guardrail metrics”, other measures to keep you honest. Your new quantitative goal would be “maximize our primary goal metric while not making the guardrail metrics any worse.”</p><p>An example of numbers looking good is “users click more on content.” An example of a related guardrail is “user reports of bad content remain low,” or “human labeling of a sample of content stays at low levels of bad content.”</p><h3>Conclusion: persuade others</h3><p>The purpose of data science is its effect on others.</p><p>Persuade. Make recommendations. Be clear and compelling. Tell a story, with evidence.</p><p>Do what’s right. Don’t just look at numbers.</p><p>All of these help you communicate with readers, listeners, decision-makers, users.</p><h3>One step</h3><p>One full step of the data science journey:</p><ol><li>Pick a good problem</li><li>Try to solve it</li><li>Communicate</li></ol><p>If you’re lucky, affect others, repeat.</p><p>Yes, luck plays a part. My efforts over the years have had widely varying levels of success, and it’s hard to tell what will happen beforehand. Just keep taking shots, and some will land.</p><p>This concludes my three-part series on how to become a data scientist. I hope you enjoyed it. Let me know.</p><p>Previous article: <a href="https://towardsdatascience.com/how-to-become-a-data-scientist-part-2-try-to-solve-the-problem-f9b9746f6eb9">How to become a data scientist, part 2: try to solve the problem</a>.</p><p>Start of the series: <a href="https://medium.com/@dfrankow/how-to-become-a-data-scientist-part-0-introduction-3fc18c2073ee">How to become a data scientist, part 0: introduction</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87f4edf1d896" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What if Recommendation Algorithms Like Facebook’s Grappled Directly with Bad Content?]]></title>
            <link>https://medium.com/@dfrankow/what-if-recommendation-algorithms-like-facebooks-grappled-directly-with-bad-content-c2f5206ec52e?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2f5206ec52e</guid>
            <category><![CDATA[organizational-culture]]></category>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Thu, 07 Oct 2021 13:46:01 GMT</pubDate>
            <atom:updated>2021-10-07T13:48:09.903Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Maybe we can improve recommendations by giving our models a different goal</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*UptMg4HE_TfAXzP3-Mdfkw.jpeg" /></figure><p>Recommendation algorithms (a.k.a recommender systems) are computer programs that choose which content to present to people. The Facebook news feed <a href="https://tech.fb.com/news-feed-ranking/">is powered by</a> a recommendation algorithm.</p><p>I first worked on recommender systems in 1997 at Net Perceptions, a Twin Cities startup that was a pioneer in the space. An early customer was Amazon, and although the business relationship turned sour, we were part of the path of developing their feature “If you like A, you’ll like B.”</p><p>I’ve also seen Silicon Valley from the inside working at or with Internet companies (Google, Amazon, Pinterest). However, I have never worked at Facebook, and I have no special or inside knowledge about their recommendation algorithms work, or about their corporate structure or internals. Given that, I’ll offer some opinions I think may be relevant.</p><h3>The goal</h3><p>Every recommender is a model (nowadays called a machine learning model, or AI model) <a href="https://towardsdatascience.com/humans-choose-ai-does-not-6512d49f7c46">formed with <strong>a mathematically well-defined goal</strong></a>. At a high level, that goal requires a dataset labeled with at least good examples (what you want to recommend) and the rest (not good, or maybe bad).</p><p>This is where things get complicated. What should the goal be?</p><p>Suppose you choose to optimize a model to increase engagement with the content, as measured by clicks. Voila, now you understand one reason the Internet is full of clickbait: we optimize for clicks. This is not a great goal. We actually care about more than just clicks. Companies care about whether the content is against policy, society might care if it’s divisive or reprehensible.</p><p>The goal you choose is crucially important to what is recommended.</p><h3>The teams</h3><p>Choosing a goal intersects with company organization in an interesting way. Large Internet companies are often led by people with two overriding goals: <strong>user growth</strong> and <strong>revenue</strong>. And growth has always mattered more, a lot more. That’s how they got big enough to matter. Facebook in particular has been obsessed with growth for years.</p><p>So, there are different groups within Facebook: some are associated directly with growth (there might be a “growth team”), and others are not. One of the teams that is not growth is “trust and safety”, the people tasked with making Facebook content conform to their policies (“<a href="https://transparency.fb.com/policies/community-standards">community standards</a>”).</p><p>I am guessing for years Mark Zuckerberg has had a constant eye on growth, and tried to think about trust and safety as little as possible. The news stories I’ve seen over the years seem to support this guess.</p><p>Growth fuels the company. Trust and safety tries to lessen the damage. No one funds or pays a company primarily because it has good community standards. They fund or pay because it has users.</p><p>The way I’ve seen that work out on a technical level is that <strong>sometimes there are two different teams dealing with content, and they don’t work together</strong>. One team tries to get some primary positive metric (like engagement, or Facebook’s more recent “<a href="https://www.facebook.com/business/news/news-feed-fyi-bringing-people-closer-together">Meaningful Social Interactions</a>”) to go up. A different team tries to get some negative metric (like posts that are against policy, or users posting bad content, or users seeing bad content) to go down.</p><p>These two teams might have entirely separate models, separate goals, even different infrastructure. There are some good reasons for that (the problems are not the same), but if teams are separate, the important team (growth, engagement) can ignore the goals of the less important team (trust and safety). I’ve seen it happen. People aren’t trying to be evil, but each team has their own goals.</p><p>For example, the team training a content recommender doesn’t want to recommend bad content, so they might simply remove it before training their model.</p><p>However, that ignores the problem. In fact, content that is engaging might be more likely to be bad. Facebook whistleblower Frances Haugen <a href="https://www.cbsnews.com/news/facebook-whistleblower-frances-haugen-60-minutes-polarizing-divisive-content/"><em>‘stated that some of Facebook’s own research found that “angry content” is more likely to receive engagement, something that content producers and political parties are aware of.’</em></a></p><h3>One model</h3><p>What if instead one model was forced to grapple with the tradeoff between engagement (positive) and bad content (negative). As a simple example, never remove any data from training, and give +1 for a post someone engaged with, but -10 for a post that was later removed as against policy.</p><p>This might apply our sophisticated machine learning techniques to finding the content that is engaging but not bad. However, such a model might also recommend only milquetoast content that offends no one, and severely reduce engagement. I’d love to know more.</p><p>Facebook might even already have experimented with this technique, but I’ve not seen public mention of it. If someone has, let me know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2f5206ec52e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Should you explain your predictions with SHAP or IG?]]></title>
            <link>https://medium.com/data-science/should-you-explain-your-predictions-with-shap-or-ig-9cabe218b5cc?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cabe218b5cc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[integrated-gradient]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[shap]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Tue, 13 Aug 2019 15:50:26 GMT</pubDate>
            <atom:updated>2019-08-14T00:29:52.069Z</atom:updated>
            <content:encoded><![CDATA[<p>Some of the most accurate predictive models today are black box models, meaning it is hard to really understand how they work. To address this problem, techniques have arisen to understand <strong>feature importance</strong>: for a given prediction, how important is each input feature value to that prediction? Two well-known techniques are <strong>SHapley Additive exPlanations (SHAP)</strong> and <strong>Integrated Gradients (IG)</strong>. In fact, they each represent a different <em>type</em> of explanation algorithm: a Shapley-value-based algorithm (SHAP) and a gradient-based algorithm (IG).</p><p>There is a fundamental difference between these two algorithm types. This post describes that difference. First, we need some background. Below, we review Shapley values, Shapley-value-based methods (including SHAP), and gradient-based methods (including IG). Finally, we get back to our central question: <strong>When should you use a Shapley-value-based algorithm (like SHAP) versus a gradient-based explanation explanation algorithm (like IG)?</strong></p><h3>What are Shapley values?</h3><p>The <a href="https://en.wikipedia.org/wiki/Shapley_value">Shapley value</a> (proposed by Lloyd Shapley in 1953) is a classic method to distribute the total gains of a collaborative game to a coalition of cooperating players. It is provably the only distribution with certain desirable properties (fully listed on Wikipedia).</p><p>In our case, we formulate a game for the prediction at each instance. We consider the “total gains” to be the prediction value for that instance, and the “players” to be the model features of that instance. The collaborative game is all of the model features cooperating to form a prediction value. The Shapley value <a href="https://en.wikipedia.org/wiki/Shapley_value#Efficiency">efficiency property</a> says the feature attributions should sum to the prediction value. The attributions can be negative or positive, since a feature can lower or raise a predicted value.</p><p>There is a variant called the <a href="https://en.wikipedia.org/wiki/Shapley_value#Aumann%E2%80%93Shapley_value">Aumann-Shapley value</a>, extending the definition of the Shapley value to a game with many (or infinitely many) players, where each player plays only a minor role, if the worth function (the gains from including a coalition of players) is differentiable.</p><h3>What is a Shapley-value-based explanation method?</h3><p>A Shapley-value-based explanation method tries to approximate Shapley values of a given prediction by examining the effect of removing a feature under all possible combinations of presence or absence of the other features. In other words, this method looks at function values over subsets of features like F(x1, <em>&lt;absent&gt;</em>, x3, x4, …, <em>&lt;absent&gt;, </em>…). How to evaluate a function F with one or more absent features is subtle.</p><p>For example, <a href="https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf">SHAP</a> (SHapely Additive exPlanations) estimates the model’s behavior on an input with certain features absent by averaging over samples from those features drawn from the training set. In other words, F(x1, &lt;absent&gt;, x3, …, xn) is estimated by the expected prediction when the missing feature x2 is sampled from the dataset.</p><p>Exactly how that sample is chosen is important (for example marginal versus conditional distribution versus cluster centers of background data), but I will skip the fine details here.</p><p>Once we define the model function (F) for all subsets of the features, we can apply the Shapley values algorithm to compute feature attributions. Each feature’s Shapley value is the contribution of the feature for all possible subsets of the other features.</p><p>The “kernel SHAP” method from the SHAP paper computes the Shapley values of all features simultaneously by defining a weighted least squares regression whose solution is the Shapley values for all the features.</p><p><strong>The high-level point is that all these methods rely on taking subsets of features.</strong> This makes the theoretical version exponential in runtime: for N features, there are 2^N combinations of presence and absence. That is too expensive for most N, so these methods approximate. Even with approximations, kernel SHAP can be slow. Also, we don’t know of any systematic study of how good the approximation is.</p><p>There are versions of SHAP specialized to different model architectures for speed. For example, <a href="https://arxiv.org/abs/1802.03888">Tree SHAP</a> computes all the subsets by cleverly keeping track of what proportion of all possible subsets flow down into each of the leaves of the tree. However, if your model architecture does not have a specialized algorithm like this, you have to fall back on kernel SHAP, or another naive (unoptimized) Shapley-value-based method.</p><p>A Shapley-value-based method is attractive as it only requires black box access to the model (i.e. computing outputs from inputs), and there is a version agnostic to the model architecture. For instance, it does not matter whether the model function is discrete or continuous. The downside is that exactly computing the subsets is exponential in the number of features.</p><h3>What is a gradient-based explanation method?</h3><p>A gradient-based explanation method tries to explain a given prediction by using the <a href="https://en.wikipedia.org/wiki/Gradient">gradient</a> of (i.e. change in) the output with respect to the input features. Some methods like Integrated Gradients (IG), GradCAM, and SmoothGrad literally apply the gradient operator. Other methods like DeepLift and LRP apply “discrete gradients.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*zIbdDZCGdIQh-ay6" /><figcaption><em>Figure 1 from </em><a href="https://arxiv.org/abs/1703.01365"><em>the IG paper</em></a><em>, showing three paths between a baseline (r</em>1<em> , r</em>2<em>) and an input (s</em>1<em>, s</em>2<em>). Path P</em>2<em>, used by Integrated Gradients, simultaneously moves all features from off to on. Path P</em>1<em> moves along the edges, turning features on in sequence. Other paths like P</em>1<em> along different edges correspond to different sequences. SHAP computes the expected attribution over all such edge paths like P</em>1<em>.</em></figcaption></figure><p>Let me describe <a href="https://arxiv.org/abs/1703.01365">IG</a>, which has the advantage that it tries to approximate Aumann-Shapley values, which are axiomatically justified. IG operates by considering a straight line path, in feature space, from the input at hand (e.g., an image from a training set) to a certain baseline input (e.g., a black image), and integrating the gradient of the prediction with respect to input features (e.g., image pixels) along this path.</p><p><a href="https://www.aclweb.org/anthology/P18-1176">This paper</a> explains the intuition of the IG algorithm as follows. As the input varies along the straight line path between the baseline and the input at hand, the prediction moves along a trajectory from uncertainty to certainty (the final prediction probability). At each point on this trajectory, one can use the gradient with respect to the input features to attribute the change in the prediction probability back to the input features. IG aggregates these gradients along the trajectory using a path integral.</p><p>IG (roughly) requires the prediction to be a continuous and piecewise differentiable function of the input features. (More precisely, it requires the function is continuous everywhere and the partial derivative along each input dimension satisfies Lebesgue’s integrability condition, i.e., the set of discontinuous points has measure zero.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oaybKcMgXepKCw8C" /></figure><p>Note it is important to choose a good baseline for IG to make sensible feature attributions. For example, if a black image is chosen as baseline, IG won’t attribute importance to a completely black pixel in an actual image. The baseline value should both have a near-zero prediction, and also faithfully represent a complete absence of signal.</p><p>IG is attractive as it is broadly applicable to all differentiable models, easy to implement in most machine learning frameworks (e.g., TensorFlow, PyTorch, Caffe), and computationally scalable to massive deep networks like Inception and ResNet with millions of neurons.</p><h3>When should you use a Shapley-value-based versus a gradient-based explanation method?</h3><p>Finally, the payoff! Our advice: <strong>If the model function is piecewise differentiable and you have access to the model gradient, use IG. Otherwise, use a Shapley-value-based method.</strong></p><p>Any model trained using gradient descent is differentiable. For example: neural networks, logistic regression, support vector machines. You can use IG with these. The major class of non-differentiable models is trees: boosted trees, random forests. They encode discrete values at the leaves. These require a Shapley-value-based method, like Tree SHAP.</p><p>The IG algorithm is faster than a naive Shapley-value-based method like kernel SHAP, as it only requires computing the gradients of the model output on a few different inputs (typically 50). In contrast, a Shapley-value-based method requires computing the model output on a large number of inputs sampled from the exponentially huge subspace of all possible combinations of feature values. Computing gradients of differentiable models is efficient and well supported in most machine learning frameworks. However, a differentiable model is a prerequisite for IG. By contrast, a Shapley-value-based method makes no such assumptions.</p><p>Several types of input features that look discrete (hence might require a Shapley-value-based method) actually can be mapped to differentiable model types (which let us use IG). Let us walk through one example: text sentiment. Suppose we wish to attribute the sentiment prediction to the words in some input text. At first, it seems that such models may be non-differentiable as the input is discrete (a collection of words). However, differentiable models like deep neural networks can handle words by first mapping them to a high-dimensional continuous space using <a href="https://en.wikipedia.org/wiki/Word_embedding">word embeddings</a>. The model’s prediction is a differentiable function of these embeddings. This makes it amenable to IG. Specifically, we attribute the prediction score to the embedding vectors. Since attributions are additive, we sum the attributions (retaining the sign) along the fields of each embedding vector and map it to the specific input word that the embedding corresponds to.</p><p>A crucial question for IG is: what is the baseline prediction? For this text example, one option is to use the embedding vector corresponding to empty text. Some models take fixed length inputs by padding short sentences with a special “no word” token. In such cases, we can take the baseline as the embedding of a sentence with just “no word” tokens.</p><p>For more on IG, see <a href="https://arxiv.org/abs/1703.01365">the paper</a> or <a href="https://github.com/ankurtaly/Integrated-Gradients/blob/master/howto.md">this how-to</a>.</p><h3>Conclusion</h3><p>In many cases (a differentiable model with a gradient), you can use integrated gradients (IG) to get a more certain and possibly faster explanation of feature importance for a prediction. However, a Shapley-value-based method is required for other (non-differentiable) model types.</p><p>At Fiddler, we support both SHAP and IG. (Full disclosure: Ankur Taly, a co-author of IG, works at <a href="http://fiddler.ai/">Fiddler</a>, and is a co-author of this post.) Feel free to email <a href="/service/mailto:info@fiddler.ai">info@fiddler.ai</a> for more information, or just to say hi!</p><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/08/should-you-explain-your-predictions-with-shap-or-ig/"><em>https://blog.fiddler.ai</em></a><em> on August 13, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cabe218b5cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/should-you-explain-your-predictions-with-shap-or-ig-9cabe218b5cc">Should you explain your predictions with SHAP or IG?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Causality in model explanations and in the real world]]></title>
            <link>https://medium.com/data-science/causality-in-model-explanations-and-in-the-real-world-9f57e17a9138?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f57e17a9138</guid>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Wed, 31 Jul 2019 13:59:15 GMT</pubDate>
            <atom:updated>2019-07-31T16:20:10.965Z</atom:updated>
            <content:encoded><![CDATA[<h4>You can’t always change a human’s input to see the output.</h4><p>At Fiddler Labs, we place great emphasis on model explanations being faithful to the model’s behavior. Ideally, feature importance explanations should surface and appropriately quantify all and only those factors that are causally responsible for the prediction. This is especially important if we want explanations to be legally compliant (e.g., <a href="https://gdpr-info.eu/art-22-gdpr/">GDPR, article 13 section 2f</a>, people have a right to <em>‘[information about] the existence of automated decision-making, including profiling .. and .. meaningful information about the logic involved’</em>), and actionable. Even when making post-processing explanations human-intelligible, we must preserve faithfulness to the model.</p><p>How do we differentiate between features that are correlated with the outcome, and those that cause the outcome? In other words, <strong>how do we think about the causality of a feature to a model output, or to a real-world task? </strong>Let’s take those one at a time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SKy4Kj12qGYP8VOB" /></figure><h3>Explaining causality in models is hard</h3><p>When explaining a model prediction, we’d like to quantify the contribution of each (causal) feature to the prediction.</p><p>For example, in a credit risk model, we might like to know how important income or zip code is to the prediction.</p><p>Note that zip code may be causal to a model’s prediction (i.e. changing zip code may change the model prediction) even though it may not be causal to the underlying task (i.e. changing zip code may not change the decision of whether to grant a loan). However, these two things may be related if this model’s output is used in the real-world decision process.</p><p>The good news is that since we have input-output access to the model, we can probe it with arbitrary inputs. This allows examining <strong>counterfactuals</strong>, inputs that are different from those of the prediction being explained. These counterfactuals might be elsewhere in the dataset, or they might not.</p><p><a href="https://en.wikipedia.org/wiki/Shapley_value">Shapley values</a> (a classic result from game theory) offer an elegant, axiomatic approach to quantify feature contributions.</p><p>One challenge is they rely on probing with an exponentially large set of counterfactuals, too large to compute. Hence, there are several papers on approximating Shapley values, especially for specific classes of model functions.</p><p>However, a more fundamental challenge is that when features are correlated, not all counterfactuals may be realistic. There is no clear consensus on how to address this issue, and existing approaches differ on the exact set of counterfactuals to be considered.</p><p>To overcome these challenges, it is tempting to rely on observational data. For instance, using the observed data to define the counterfactuals for applying Shapley values. Or more simply, fitting an interpretable model on it to mimic the main model’s prediction and then explaining the interpretable model in lieu of the main model. But, this can be dangerous.</p><p>Consider a credit risk model with features including the applicant’s income and zip code. Say the model internally only relies on the zip code (i.e., it <a href="https://en.wikipedia.org/wiki/Redlining">redlines</a> applicants). Explanations based on observational data might reveal that the applicant’s income, by virtue of being correlated to zip code, is as predictive of the model’s output. This may mislead us to explain the model’s output in terms of the applicant’s income. In fact, a naive explanation algorithm will split attributions equally between two perfectly correlated features.</p><p>To learn more, we can intervene in features. One counterfactual changing zip code but not income will reveal that zip code causes the model’s prediction to change. A second counterfactual that changes income but not zip code will reveal that income does not. These two together will allow us to conclude that zip code is causal to the model’s prediction, and income is not.</p><p>Explaining causality requires the right counterfactuals.</p><h3>Explaining causality in the real world is harder</h3><p>Above we outlined a method to try to explain causality in models: study what happens when features change. To do so in the real world, you have to be able to apply interventions. This is commonly called a “<a href="https://en.wikipedia.org/wiki/Randomized_controlled_trial">randomized controlled trial</a>” (also known as an “<a href="https://en.wikipedia.org/wiki/A/B_testing">A/B testing</a>” when there are two variants, especially in the tech industry). You divide a population into two or more groups randomly, and apply different interventions to each group. The randomization ensures that the only differences among the groups are your intervention. Therefore, you can conclude that your intervention causes the measurable differences in the groups.</p><p>The challenge in applying this method to real-world tasks is that not all interventions are feasible. You can’t ethically ask someone to take up smoking. In the real world, you may not be able to get the data you need to properly examine causality.</p><p>We can probe models as we wish, but not people.</p><p><a href="https://en.wikipedia.org/wiki/Natural_experiment">Natural experiments</a> can provide us an opportunity to examine situations where we would not normally intervene, like in epidemiology and economics. However, these provide us a limited toolkit, leaving many questions in these fields up for debate.</p><p>There are proposals for <a href="https://en.wikipedia.org/wiki/Causality#Theories">other theories</a> that allow us to use domain knowledge to separate correlation from causation. These are subject to ongoing debate and research.</p><p>Now you know why explaining causality in models is hard, and explaining it in the real world is even harder.</p><p>To learn more about explaining models, email us at <a href="/service/mailto:info@fiddler.ai">info@fiddler.ai</a>. (Photo credit: <a href="https://pixabay.com/illustrations/mikado-domino-stones-pay-steinchen-1013877/">pixabay</a>.) This post was co-written with Ankur Taly.</p><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/07/causality-in-model-explanations-and-in-the-real-world/"><em>https://blog.fiddler.ai</em></a><em> on July 31, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f57e17a9138" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/causality-in-model-explanations-and-in-the-real-world-9f57e17a9138">Causality in model explanations and in the real world</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Hey, what’s that?” Debugging predictions using explanations]]></title>
            <link>https://medium.com/data-science/https-towardsdatascience-com-that-debugging-predictions-using-explanations-861cc94cb875?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/861cc94cb875</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Mon, 22 Jul 2019 16:26:36 GMT</pubDate>
            <atom:updated>2019-07-22T16:39:15.190Z</atom:updated>
            <content:encoded><![CDATA[<p>Machine learning (ML) models are popping up everywhere. There is a lot of technical innovation (e.g., deep learning, explainable AI) that has made them more accurate, more broadly applicable, and usable by more people in more business applications. The lists are everywhere: <a href="https://www.techradar.com/news/how-ai-and-machine-learning-our-improving-the-banking-experience">banking</a>, <a href="https://www.accenture.com/us-en/insight-artificial-intelligence-healthcare">healthcare</a>, <a href="https://www.wired.com/story/guide-self-driving-cars/">tech</a>, <a href="https://www.forbes.com/sites/forbestechcouncil/2018/09/27/15-business-applications-for-artificial-intelligence-and-machine-learning/#413d9992579f">all of the above</a>.</p><p>However, as with any computer program, models can have errors, or more colloquially, <a href="https://web.archive.org/web/20000818180803/http://www.jamesshuggins.com/h/tek1/first_computer_bug_large.htm">bugs</a>. The process of finding those bugs is quite different from previous technology, and <a href="https://blog.fiddler.ai/2019/06/ai-needs-a-new-developer-stack/">requires a new developer stack</a>. “ <a href="https://www.wired.com/2016/05/the-end-of-code/">Soon we won’t program computers, we’ll train them like dogs</a>” (Wired, 2016). “ <a href="https://medium.com/@karpathy/software-2-0-a64152b37c35">Gradient descent can write code better than you. I’m sorry</a> “ (Andrej Karpathy, 2017).</p><p>In a deep learning neural network, instead of lines of code written by people, we are looking at possibly millions of weights linked together into an incomprehensible network. ( <a href="https://commons.wikimedia.org/wiki/File:Two-layer_feedforward_artificial_neural_network.png">picture credit</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uqphh22Ys1Y6ig2a" /><figcaption>How do we find bugs in this network?</figcaption></figure><p>So how do we find bugs in this network? One way is to explain your model predictions. Let’s look at two types of bugs we can find through explanations (data leakage and data bias), illustrated with examples from predicting loan default. Both of these are actually data bugs, but a model summarizes the data, so they show up in the model.</p><p>Most ML models are supervised. You <a href="https://blog.fiddler.ai/2019/05/humans-choose-ai-does-not/">choose</a> a precise prediction goal (also called the “prediction target”), gather a dataset with features, and label each example with the target. Then you train a model to use the features to predict the target. Surprisingly often there are features in the dataset that relate to the prediction target but are not useful for prediction. For example, they might be added from the future (i.e. long after prediction time), or otherwise unavailable at prediction time.</p><p>Here is an example from the <a href="https://github.com/fiddler-labs/p2p-lending-data">Lending Club dataset</a>. We can use this dataset to try modeling predicting loan default with loan_status field as our prediction target. It takes the values “Fully Paid” (okay) or “Charged Off” (bank declared a loss, i.e. the borrower defaulted). In this dataset, there are also fields such as total_pymnt (the payments received) and loan_amnt (amount borrowed). Here are a few example values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/0*1R5Vsm6j_uxmOT-E" /><figcaption><em>Whenever the loan is “Charged Off”, delta is positive. But, we don’t know delta at loan time.</em></figcaption></figure><p>Notice anything? Whenever the loan has defaulted (“Charged Off”), the total payments are less than the loan amount, and delta (=loan_amnt-total_pymnt) is positive. Well, that’s not terribly surprising. Rather, it’s nearly the definition of default: by the end of the loan term, the borrower paid less than what was loaned. Now, delta doesn’t <em>have</em> to be positive for a default: you could default after paying back the entire loan principal amount but not all of the interest. But, in this data, 98% of the time if delta is negative, the loan was fully paid; and 100% of the time delta is positive, the loan was charged off. Including total_pymnt gives us nearly perfect information, but we don’t get total_pymnt until after the entire loan term (3 years)!</p><p>Including both loan_amnt and total_pymnt in the data potentially allows nearly perfect prediction, but we won’t really have total_pymnt for the real prediction task. Including them both in the training data is <em>data leakage</em> of the prediction target.</p><p>If we make a (cheating) model, it will perform very well. Too well. And, if we run a feature importance algorithm on some predictions (a common form of model explanation), we’ll see these two variables come up as important, and with any luck realize this data leakage.</p><p>Below, the Fiddler explanation UI shows “delta” stands out as a huge factor in raising this example prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mAfxFpzw2hXb-OwX" /><figcaption>“delta” really stands out, because it’s data leakage.</figcaption></figure><p>There are other, more subtle potential data leakages in this dataset. For example, the grade and sub_grade are <a href="https://www.lendingclub.com/foliofn/rateDetail.action">assigned</a> by a Lending Club proprietary model, which almost completely determines the interest rate. So, if you want to build your own risk scoring model without Lending Club, then grade, sub_grade, and int_rate are all data leakage. They wouldn’t allow you to perfectly predict default, but presumably they would help, or Lending Club would not use their own model. Moreover, for their model, they include <a href="https://www.fico.com/en/products/fico-score">FICO score</a>, yet another proprietary risk score, but one that most financial institutions buy and use. If you don’t want to use FICO score, then that is also data leakage.</p><p>Data leakage is any predictive data that you can’t or won’t use for prediction. A model built on data with leakage is buggy.</p><p>Suppose through poor data collection or a bug in preprocessing, our data in biased. More specifically, there is a spurious correlation between a feature and the prediction target. In that case, explaining predictions will show an unexpected feature often being important.</p><p>We can simulate a data processing bug in our lending data by dropping all the charged off loans from zip codes starting with 1 through 5. Before this bug, zip code is not very predictive of chargeoff (an <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve">AUC</a> of 0.54, only slightly above random). After this bug, any zip code starting with 1 through 5 will never be charged off, and the AUC jumps to 0.78. So, zip code will show up as an important feature in predicting (no) loan default from data examples in those zip codes. In this example, we could investigate by looking at predictions where zip code was important. If we are observant, we might notice the pattern, and realize the bias.</p><p>Below is what charge-off rate would look like if summarized by the first digit of zip code. Some zips would have no charge-offs, while the rest had a rate similar to the dataset overall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*xhaUsHfOELLbrFEz" /><figcaption>In this buggy dataset, there are no charged-off loans with zip codes starting with 6, 7, 8, 9, 0.</figcaption></figure><p>Below, the Fiddler explanation UI shows zip code prefix stands out as a huge factor in lowering this example prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EPxQDzRjtwK_GBFu" /><figcaption>“zip_code_prefix” really stands out, because the model has a bug related to zip code.</figcaption></figure><p>A model built from this biased data is not useful for making predictions on (unbiased) data we haven’t seen yet. It is only accurate in the biased data. Thus, a model built on biased data is buggy.</p><p>There are many other possibilities for model debugging that don’t involve model explanations. For example:</p><ol><li>Look for <a href="https://en.wikipedia.org/wiki/Overfitting">overfitting</a> or underfitting. If your model architecture is too simple, it will underfit. If it is too complex, it will overfit.</li><li>Regression tests on a golden set of predictions that you understand. If these fail, you might be able to narrow down which scenarios are broken.</li></ol><p>Since explanations aren’t involved with these methods, I won’t say more here.</p><p>If you are not sure your model is using your data appropriately, use explanations of feature importance to examine its behavior. You might see data leakage or data bias. Then, you can fix your data, which is the best way to fix your model.</p><p>Fiddler is building an <a href="https://fiddler.ai/">Explainable AI Engine</a> that can help debug models. Email us at <a href="/service/mailto:info@fiddler.ai">info@fiddler.ai</a>.</p><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/07/debugging-predictions-using-explanations/?preview=true"><em>https://blog.fiddler.ai</em></a><em> on July 22, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=861cc94cb875" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/https-towardsdatascience-com-that-debugging-predictions-using-explanations-861cc94cb875">“Hey, what’s that?” Debugging predictions using explanations</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A gentle introduction to GA2Ms, a white box model]]></title>
            <link>https://medium.com/data-science/a-gentle-introduction-to-ga2ms-a-white-box-model-d05d13c6ca00?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d05d13c6ca00</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[white-box-model]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Mon, 03 Jun 2019 13:50:04 GMT</pubDate>
            <atom:updated>2019-08-19T17:19:39.388Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is a gentle introduction to a white box machine learning model called a GA2M.</p><p>We’ll walk through:</p><ul><li>What is a white box model, and why would you want one?</li><li>A classic example white box model: logistic regression</li><li>What is a GAM, and why would you want one?</li><li>What is a GA2M, and why would you want one?</li><li>When should you choose a GAM, a GA2M, or something else?</li></ul><p>The purpose of all these machine learning models is to make a prediction towards <a href="https://blog.fiddler.ai/2019/05/humans-choose-ai-does-not/">a goal specified by a human</a>. Think of a model that can predict loan default, or the presence of someone’s face in a picture.</p><p>The short story: A generalized additive model (GAM) is a white box model that is more flexible than logistic regression, but still interpretable. A GA2M is a GAM with interaction terms, which allows it to be more flexible still, but with a more complicated interpretation. GAMs and GA2Ms are an intriguing addition to your toolbox, interpretable at the expense of not fitting every kind of data. A picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OloNK-yzwYTgZYHj.png" /></figure><p>For more about what that all means, read on.</p><h3>White box models</h3><p>The term “ <a href="https://en.wikipedia.org/wiki/White_box_%28software_engineering%29">white box</a>” comes from software engineering. It means software whose internals you can view, compared to a “black box” whose internals you cannot view. By this definition, a <a href="https://en.wikipedia.org/wiki/Artificial_neural_network">neural network</a> could be a white box model if you can see the weights (<a href="https://commons.wikimedia.org/wiki/File:Two-layer_feedforward_artificial_neural_network.png">picture credit</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XE-lJbGtd1XC_x5E.png" /></figure><p>However, by white box people really mean something they can <em>understand</em>. <strong>A white box model is a model whose internals a person can see and reason about</strong>. This is subjective, but most people would agree the weights shown above don’t give us information about how the model works in such a way as we could usefully describe it, or predict what the model is going to do in the future.</p><p>Compare the picture above to this one about risk of death from pneumonia by age from [1]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*neMWK_JKvNB25MBQ.png" /></figure><p>Now that isn’t a whole model. Rather, it’s the impact of one feature (age) on the risk score. The green lines are error bars (±1 standard deviation in 100 rounds of <a href="https://en.wikipedia.org/wiki/Bootstrap_aggregating">bagging</a>). The red line in the middle of them is the best estimate. In the paper, they observe:</p><ol><li>Risk score is flat until about age 50. Risk score here is negative, meaning less risk of death than the average in the dataset.</li><li>Risk score rises sharply at 65. This could be due to retirement. In future, it might be interesting to gather data about retirement.</li><li>The error bars are narrowest in ages 66–85. Perhaps that is where the most data is.</li><li>Risk score rises again at 85. The error bars also widen again. Maybe this jump is not real.</li><li>Risk score drops above 100. This may be due to lack of data, or something else. In the paper, they suggested one might wish to “fix” this region of the model by changing it to predict at the same level as ages 85–100 instead of dropping. This fix is using domain knowledge (“risk of pneumonia likely doesn’t go down after age 85”) to address possible model artifacts.</li><li>Risk score between 66 and 85 is relatively flat.</li></ol><p>All this from one graph of one model feature. There are facts, like the shape of the graph, and then speculation about why the graph might behave that way. The facts are useful to understand the data. The speculation cannot be answered by any tool, but may be useful to suggest further actions, like collecting new features (say, about retirement) or new instances (like points below age 50 or above 100), or new analyses (like looking carefully at data instances around ages 85–86 for differences).</p><p>These aren’t simulations of what the model would do. These are the internals of the model itself, so that graph is accurately describing the exact effect of age on risk score. There are 55 other components to this model, but each can be examined and reasoned about.</p><p>This is the power of a white box model.</p><p>This example also shows the dangers. By seeing everything, we may believe we understand everything, and speculate wildly or “fix” inappropriately. As always, we have to exercise judgment to use data properly.</p><p>In summary: make a white box model to</p><ul><li><strong>learn about your model</strong>, not from simulations or approximations, but the actual internals</li><li><strong>improve your model</strong>, by giving you ideas of directions to pursue</li><li><strong>“fix” your model</strong>, i.e., align it with your intuition or domain knowledge</li></ul><p>One final possibility: regulations dictate that you need to fully describe your model. In that case, it could be useful to have human-readable internals for reference.</p><p>Here are some examples of white box and black box models:</p><p>White box models</p><ul><li>Logistic regression</li><li>GAMs</li><li>GA2Ms</li><li>Decision trees (short and few trees)</li></ul><p>Black box models</p><ul><li>Neural networks (including deep learning)</li><li>Boosted trees and random forests (many trees)</li><li>Support vector machines</li></ul><p>Now let’s walk through three specific white box models.</p><h3>A classic: logistic regression</h3><p><a href="https://en.wikipedia.org/wiki/Logistic_regression#History">Logistic regression</a> was developed in the early 1800s, and re-popularized in the 1900s. It’s been around for a long time, for many reasons. It solves a common problem (predict the probability of an event), and it’s interpretable. Let’s explore what that means. Here is the logistic equation defining the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F6m3Oj9kzi3DHiNB.png" /></figure><p>There are three types of variables in this model equation:</p><ul><li>p is the probability of an event we’re predicting. For example, defaulting on a loan</li><li>The x’s are features. For example, loan amount.</li><li>The 𝛽’s (betas) are the coefficients, which we fit using a computer.</li></ul><p>The betas are fit once to the entire dataset. The x’s are different for each instance in the dataset. The p represents an aggregate of dataset behavior: any dataset instance either happened (1) or didn’t (0), but in aggregate, we’d like the right-hand side and the left-hand side to be as close as possible.</p><p>The “log(p/(1-p))” is the log odds, also called the “logit of the probability”. The odds are (probability the event happened)/(probability the event won’t happen), or p/(1-p). Then we apply the natural logarithm to translate p, which takes the range 0 to 1, to a quantity which can range from -∞ to +∞, suitable for a linear model.</p><p>This model is linear, but for the log odds. That is, the right-hand side is a linear equation, but it is fit to the log odds, not the probability of an event.</p><p>This model is interpretable as follows: <strong>a unit increase in xi is a log-odds increase in 𝛽i</strong>.</p><p>For example, suppose we’re predicting probability of loan default, and our model has a feature coefficient 𝛽1=0.15 for the loan amount feature x1. That means a unit increase in the feature corresponds to a log odds increase of 0.15 in default. We can take the natural exponent to get the odds ratio, exp(0.15)=1.1618. That means:</p><p><em>for this model, a unit increase (of say, a thousand dollars) in loan amount corresponds to a 16% increase in the odds of loan default, holding all other factors constant.</em></p><p>This statement is what people mean when they say logistic regression is interpretable.</p><p>To summarize why logistic regression is a white box model:</p><ul><li>The input response terms (𝛽i*xi terms) can be interpreted independently of each other</li><li>The terms are in interpretable units: the coefficients (betas) are in units of log odds.</li></ul><p>So why would we use anything other than the friendly, venerable model of logistic regression?</p><p>Well, if the features and log odds don’t have a linear relationship, this model won’t fit well. I always think of trying to fit a line to a parabola:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*U3CJVH_gaocbCvGg.png" /><figcaption>No line fits a curve.</figcaption></figure><p>If you have non-linear data (the black parabola), a linear fit (the blue dashed line) will never be great. No line fits the curve.</p><h3>Generalized Additive Models (GAMs)</h3><p><a href="https://en.wikipedia.org/wiki/Generalized_additive_model">Generalized Additive Models</a> (GAMs) were developed in the 1990s by Hastie and Tibshirani. (See also chapter 9 of their book “ <a href="https://web.stanford.edu/~hastie/ElemStatLearn/">The Elements of Statistical Learning</a> “.) Here is the equation defining the model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pq5vs0MuZFyzg_Pw.png" /></figure><p>This equation is quite similar to logistic regression. It has the same three types of elements:</p><ul><li>E(Y) is an aggregate of dataset behavior, like the “p” in the equation above. In fact, it may well be the probability of an event, the same p.</li><li>g(.) is a link function, like the logit (or log odds) from the logistic equation above.</li><li>fi(xi) is a term for each dataset instance feature x1,…,xm.</li></ul><p>The big difference is instead of a linear term 𝛽i*xi for a feature, now we have a function fi(xi). In their book, Hastie and Tibshirani specify a “smooth” function like a cubic <a href="https://en.wikipedia.org/wiki/Spline_(mathematics)">spline</a>. Lou et al. [2] looked at other functions for the fi, which they call “shape functions.”</p><p>A GAM also has white box features:</p><ul><li>The input response terms (f(xi) terms) can be interpreted independently of each other</li><li>The terms are in interpretable units. For the logit link function, these are log odds.</li></ul><p>Now a term, instead of being a constant (beta), is a function, so instead of reporting the log odds as a number, we visualize it with a graph. In fact, the graph above of pneumonia risk of death by age is one term (shape function) in a GAM.</p><p>So why would we use anything other than a GAM? It’s already flexible and interpretable. Same reason as before: it might not be accurate enough. In particular, we’ve assumed that each feature response can be modeled with its own function, independent of the others.</p><p>But what if there are interactions between the features? Several black box models (boosted trees, neural networks) can model interaction terms. Let us walk through a white box model that also can: GA2Ms.</p><h3>GAMs with interaction terms (GA2Ms)</h3><p>GA2Ms were investigated in 2013 by Lou et al. [3]. The authors pronounce them with the letters “gee ay two em”, but in house we’ve taken to calling them “interaction GAMs” because it’s more pronounceable. Here is the model equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_d2XLSYFQdbzxhIP.png" /></figure><p>This equation is quite similar to the GAM equation from the previous section, except it adds functions that can account for two feature variables at once, i.e. interaction terms.</p><p>Microsoft just released a library <a href="https://github.com/microsoft/interpret">InterpretML</a> that implements GA 2Ms in python. In that library, they call them “Explainable Boosting Machines.”</p><p>Lou et al. say these are still white box models because the “shape function” for an interaction term is a heatmap. The two features are along the X and Y axis, and the color in the middle shows the function response. Here is an example from Microsoft’s library fit to predicting loan default on <a href="https://github.com/fiddler-labs/p2p-lending-data/">a dataset of loan performance from lending club</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ITjH4S3qv9oqXnIi.png" /></figure><p>For this example graph:</p><ul><li>The upper right corner is the most red. That means the probability of default goes up the most when dti (debt to income ratio) and fico_range_midpoint (the FICO credit score) are both high.</li><li>The left strip is also red, but turns blue near the bottom. That means that very low dti is usually bad, except if fico_range_midpoint is also low.</li></ul><p>This particular heatmap is hard to reason about. This is likely only the interaction effect without the single-feature terms. So, it could be that the probability of default overall isn’t higher at high-dti and high-fico, but rather just higher than either of the primary effects predict by themselves. To investigate further, we could probably look at some examples around the borders. But, for this blog post, we’ll skip the deep dive.</p><p>In practice, this library fits all single-feature functions, then N interaction terms, where you pick N. It is not easy to pick N. The interaction terms are worthwhile if they add enough accuracy to be worth the extra complexity of staring at heatmaps to interpret them. That is a judgement call that depends on your business situation.</p><h3>When should we use GAMs or GA2Ms?</h3><p>To perform machine learning, first <a href="https://blog.fiddler.ai/2019/05/humans-choose-ai-does-not/">pick a goal</a>. Then pick a technology that will best use your data to meet the goal. There are thousands of books and millions of papers on that subject. But, here is a drastically simplified way to think about how GA2Ms fit in to possible model technologies: they are on a spectrum from interpretability to modeling feature interactions.</p><ul><li>Use GAMs if they are accurate enough. It gives the advantages of a white box model: separable terms with interpretable units.</li><li>Use GA2Ms if they are significantly more accurate than GAMs, especially if you believe from your domain knowledge that there are real feature interactions, but they are not too complex. This also gives the advantages of a white box model, with more effort to interpret.</li><li>Try boosted trees (xgboost or lightgbm) if you don’t know a lot about the data, since it is quite robust to quirks in data. These are black box models.</li><li>When features interact highly with each other, like pixels in images or the context in audio, you may well need neural networks or something else that can capture complex interactions. These are deeply black box.</li></ul><p>In all cases, you may well need domain-specific data preprocessing, like squaring images, or standardizing features (subtracting the mean and dividing by the standard deviation). That is a topic for another day.</p><p>Now hopefully the diagram we started with makes more sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*044NBj7G_iM6MrlI.png" /></figure><p>At <a href="http://fiddler.ai/">Fiddler Labs</a>, we help you explain your AI. Email us at <a href="/service/mailto:info@fiddler.ai">info@fiddler.ai</a>.</p><h3>References</h3><ol><li>Caruana, Rich, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. “Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-Day Readmission.” In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721–1730. KDD ’15. New York, NY, USA: ACM, 2015. <a href="https://doi.org/10.1145/2783258.2788613">https://doi.org/10.1145/2783258.2788613</a>.</li><li>Lou, Yin, Rich Caruana, and Johannes Gehrke. “Intelligible Models for Classification and Regression.” In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 150–158. KDD ’12. New York, NY, USA: ACM, 2012. <a href="https://doi.org/10.1145/2339530.2339556">https://doi.org/10.1145/2339530.2339556</a>.</li><li>Lou, Yin, Rich Caruana, Giles Hooker, and Johannes Gehrke. Accurate Intelligible Models with Pairwise Interactions, 2017. <a href="https://www.microsoft.com/en-us/research/publication/accurate-intelligible-models-pairwise-interactions/">https://www.microsoft.com/en-us/research/publication/accurate-intelligible-models-pairwise-interactions/</a>.</li></ol><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/06/a-gentle-introduction-to-ga2ms-a-white-box-model/"><em>https://blog.fiddler.ai</em></a><em> on June 3, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d05d13c6ca00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-gentle-introduction-to-ga2ms-a-white-box-model-d05d13c6ca00">A gentle introduction to GA2Ms, a white box model</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Humans choose, AI does not]]></title>
            <link>https://medium.com/data-science/humans-choose-ai-does-not-6512d49f7c46?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6512d49f7c46</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Wed, 08 May 2019 17:31:09 GMT</pubDate>
            <atom:updated>2019-05-08T20:38:51.367Z</atom:updated>
            <content:encoded><![CDATA[<h3>Artificial intelligence isn’t human</h3><p>“<a href="https://www.newsweek.com/artificial-intelligence-will-take-our-jobs-2060-618259">Artificial Intelligence Will Best Humans at Everything by 2060, Experts Say</a>”. Well.</p><p>First, as Yogi Berra said, “It’s tough to make predictions, especially about the future.” Where is my flying car?</p><p>Second, the title reads like clickbait, but surprisingly it appears to be pretty close to the <a href="https://arxiv.org/abs/1705.08807">actual survey</a>, which asked AI researchers when “high-level machine intelligence” will arrive, defined as <em>“when unaided machines can accomplish every task better and more cheaply than human workers.”</em> What is a ‘task’ in this definition? Does “every task” even make sense? Can we enumerate all tasks?</p><p>Third and most important, is high-level intelligence just accomplishing tasks? This is the real difference between artificial and human intelligence: humans define goals, AI tries to achieve them. Is the hammer going to displace the carpenter? They each have a purpose.</p><p>This difference between artificial and human intelligence is crucial to understand, both to interpret all the crazy headlines in the popular press, and more importantly, to make practical, informed judgements about the technology.</p><p>The rest of this post walks through some types of artificial intelligence, types of human intelligence, and given how different they are, plausible and implausible risks of artificial intelligence. The short story: unlike humans, <em>every</em> AI technology has a perfectly mathematically well-defined goal, often a labeled dataset.</p><h3>Types of artificial intelligence</h3><p>In <strong>supervised learning</strong>, you define a prediction goal and gather a training set with labels corresponding to the goal. Suppose you want to identify whether a picture has Denzel Washington in it. Then your training set is a set of pictures, each labeled as containing Denzel Washington or not. The label has to be applied outside of the system, mostly likely by people. If your goal is to do facial recognition, your labeled dataset is pictures along with a label (the person in the picture). Again, you have to gather the labels somehow, likely with people. If your goal is to match a face with another face, you need a label of whether the match was successful or not. Always labels.</p><p>Almost all the machine learning you read about is supervised learning. Deep learning, neural networks, decision trees, random forests, logistic regression, all training on labeled datasets.</p><p>In <strong>unsupervised learning</strong>, again you define a goal. A very common unsupervised learning technique is clustering (e.g., the well-known <a href="https://en.wikipedia.org/wiki/K-means_clustering">k-means clustering</a>). Again, the goal is very well-defined: find clusters minimizing some mathematical cost function. For example, where the distance between points within the same cluster is small, and the distance between points not within the same cluster is large. All of these goals are so well-defined they have mathematical formalism:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/0*kMRsxKh1agvXPp8G" /></figure><p>This formula feels very different from how humans specify goals. Most humans don’t understand these symbols at all. They are not formal. Also, a “goal-oriented” mindset in a human is unusual enough that it has a special term.</p><p>In <strong>reinforcement learning</strong>, you define a reward function to reward (or penalize) actions that move towards a goal. This is the technology people have been using recently for games like chess and Go, where it may take many actions to reach a particular goal (like checkmate), so you need a reward function that gives hints along the way. Again, not only a well-defined goal, but even a well-defined on-the-way-to-goal reward function.</p><p>These are types of artificial intelligence (“machine learning”) that are currently hot because of recent huge gains in accuracy, but there are plenty of others that people have studied.</p><p><strong>Genetic algorithms</strong> are another way of solving problems inspired by biology. One takes a population of mathematical constructs (essentially functions), and selects those that perform best on a problem. Although people get emotional about the biological analogy, still the fitness function that defines “best” is a concrete, completely specified mathematical function chosen by a human.</p><p>There is <strong>computer-generated art</strong>. For example, <a href="https://github.com/google/deepdream">deep dream</a> (<a href="https://photos.google.com/share/AF1QipPX0SCl7OzWilt9LnuQliattX4OUCj_8EP65_cTVnBmS1jnYgsGQAieQUc1VQWdgQ?key=aVBxWjhwSzg2RjJWLWRuVFBBZEN1d205bUdEMnhB">gallery</a>) is a way to generate images from deep learning neural networks. This would seem to be more human and less goal-oriented, but in fact people are still directing. The authors described the goal at a high level: “Whatever you see there, I want more of it!” Depending on which layer of the network is asked, the features amplified might be low level (like lines, shapes, edges, colors, see the addaxes below) or higher level (like objects).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DovZ-SeTzUmO_XJv.png" /><figcaption><a href="https://www.flickr.com/photos/zachievenor/8258092492/in/set-72157630014410078"><em>Original photo</em></a><em> of addaxes by Zachi Evenor, </em><a href="https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html"><em>processed photo</em></a><em> from Google</em></figcaption></figure><p><strong>Expert systems</strong> are a way to make decisions using if-then rules on a formally expressed body of knowledge. They were somewhat popular in the 1980s. These are a type of “<a href="https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence">Good Old Fashioned Artificial Intelligence</a>” (GOFAI), a term for AI based on manipulating symbols.</p><p>Another common difference between human and artificial intelligence is that humans learn over a long time, while AI is often retrained from the beginning for each problem. This difference, however, is being narrowed. <strong>Transfer learning</strong> is the process of training a model, and then using or tweaking the model for use in a different context. This is industry practice in computer vision, where deep learning neural networks that have been trained using features from previous networks (<a href="https://labs.pinterest.com/user/themes/pin_labs/assets/paper/visual_search_at_pinterest.pdf">example</a>).</p><p>One interesting research project in long-term learning is <a href="http://rtw.ml.cmu.edu/rtw/">NELL</a>, Never-Ending Language Learning. NELL crawls the web collecting text, and trying to extract beliefs (facts) like “<a href="http://rtw.ml.cmu.edu/rtw/kbbrowser/company:airtran">airtran</a> is a <a href="http://rtw.ml.cmu.edu/rtw/kbbrowser/pred:transportation">transportation system</a>”, along with a confidence. It’s been crawling since 2010, and as of July 2017 has <a href="https://cacm.acm.org/magazines/2018/5/227193-never-ending-learning/fulltext">accumulated</a> over 117 million candidate beliefs, of which 3.8 million are high-confidence (at least 0.9 of 1.0).</p><p>In every case above, humans not only specify a goal, but have to specify it unambiguously, often even formally with mathematics.</p><h3>Types of human intelligence</h3><p>What are the types of human intelligence? It’s hard to even come up with a list. Psychologists have been studying this for decades. Philosophers have been wrestling with it for millennia.</p><p><a href="https://en.wikipedia.org/wiki/Intelligence_quotient">IQ</a> (the Intelligence Quotient) is measured with verbal and visual tests, sometimes abstract. It is predicated on the idea that there is a general intelligence (sometimes called the “<a href="https://en.wikipedia.org/wiki/G_factor_(psychometrics)">g factor</a>”) common to all cognitive ability. This idea is not accepted by everyone, and IQ itself is hotly debated. For example, some believe that people with the same latent ability but from different demographic groups might be measured differently, called <a href="https://en.wikipedia.org/wiki/Intelligence_quotient#Test_bias_or_differential_item_functioning">Differential Item Functioning</a>, or simply measurement bias.</p><p>People <a href="https://en.wikipedia.org/wiki/Fluid_and_crystallized_intelligence">describe</a> fluid intelligence (the ability to solve novel problems) and crystallized intelligence (the ability to use knowledge and experience).</p><p>The concept of <a href="https://en.wikipedia.org/wiki/Emotional_intelligence">emotional intelligence</a> shows up in the popular press: the ability of a person to recognize their own emotions and those of others, and use emotional thinking to guide behavior. It is unclear how accepted this is by the academic community.</p><p>More widely accepted are the <a href="https://en.wikipedia.org/wiki/Big_Five_personality_traits">Big Five personality traits</a>: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. This is not intelligence (or is it?), but illustrates a strong difference with computer intelligence. “Personality” is a set of stable traits or behavior patterns that predict a person’s behavior. What is the personality of an artificial intelligence? The notion doesn’t seem to apply.</p><p>With humor, art, or the search for meaning, we get farther and farther from well-defined problems, yet closer and closer to humanity.</p><h3>Risks of artificial intelligence</h3><p>Can artificial intelligence surpass human intelligence?</p><p>One risk that captures the popular press is <strong>The Singularity</strong>. The writer and mathematician Verner Vinge gives a compelling description in <a href="http://mindstalk.net/vinge/vinge-sing.html">an essay</a> from 1993: “<em>Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.”</em></p><p>There are at least two ways to interpret this risk. The common way is that some magical critical mass will cause a phase change in machine intelligence. I’ve never understood this argument. “More” doesn’t mean “different.” The argument is something like “As we mimic the human brain closely, something near-human (or super-human) will happen.” Maybe?</p><p>Yes, the availability of lots of computing power and lots of data has resulted in a phase change in AI <em>results</em>. Speech recognition, automatic translation, computer vision, and other problem domains have been completely transformed. In 2012, when researchers at Toronto re-applied neural networks to computer vision, the error rates on a well-known dataset started dropping fast, until within a few years the computers were beating the humans. <strong>But computers were doing the same well-defined task as before, only better.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q2jkML4txNsg3K-R.png" /><figcaption><a href="https://arxiv.org/abs/1409.0575"><em>ImageNet Large Scale Visual Recognition Challenge</em></a><em> error rates. 0.1 is a 10% error rate. After neural networks were re-applied in 2012, error rates dropped fast. They </em><a href="http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/"><em>beat</em></a><em> Andrej Karpathy by 2015.</em></figcaption></figure><p>The more compelling observation is: “The chance of a singularity might be small, but the consequences are so serious we should think carefully.”</p><p>Another way to interpret the risk of the singularity is that the entire system will have a phase change. That system includes computers, technology, networks, and humans setting goals. This seems more correct and entirely plausible. The Internet was a phase change, as were mobile phones. Under this interpretation, there are plenty of plausible risks of AI.</p><p>One plausible risk is <strong>algorithmic bias</strong>. If algorithms are involved in important decisions, we’d like them to be trustworthy. (In a <a href="https://blog.fiddler.ai/2019/04/a-gentle-introduction-to-algorithmic-fairness/">previous post</a>, we discussed how to measure algorithmic fairness.)</p><p>Tay, a Microsoft chatbot, was <a href="https://arstechnica.com/information-technology/2016/03/tay-the-neo-nazi-millennial-chatbot-gets-autopsied/">taught</a> by Twitter to be racist and woman-hating within 24 hours. But Tay didn’t really understand anything, it just “learned” and mimicked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T5WO9VnHX6QPlLRA.png" /></figure><p>Amazon’s facial recognition software Rekognition, falsely <a href="https://www.cnet.com/news/amazon-facial-recognition-thinks-28-congressmen-look-like-known-criminals-at-default-settings/">matched</a> 28 U.S. Congresspeople (mostly people of color) with known criminals. Amazon’s response was that the ACLU (who conducted the test) used an unreliable cutoff of only 80 percent confident. (They recommended 95 percent.)</p><p>MIT researcher Joy Buolamwini <a href="https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html">showed</a> that gender identification error rates in several computer vision systems were much higher for people with dark skin.</p><p>All of these untrustworthy results arise at least partially from the training data. In Tay’s case, it was deliberately fed hateful data by Twitter users. In the computer vision systems, there may well have been less data for people of color.</p><p>Another plausible risk is <strong>automation</strong>. As artificial intelligence becomes more cost-efficient at solving problems like driving cars or weeding farm plots, the people who used to do those tasks may be thrown out of work. This is the risk of AI plus capitalism: businesses will each try to be cost effective. We can only address this at a societal level, which makes it very difficult.</p><p>One final risk is <strong>bad goals</strong>, possibly aggravated by single-mindedness. This is memorably illustrated by the paper-clip problem, first <a href="https://nickbostrom.com/ethics/ai.html">described</a> by Nick Bostrom in 2003: <em>“It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal. For better or worse, artificial intellects need not share our human motivational tendencies.”</em> There is even a <a href="http://www.decisionproblem.com/paperclips/index2.html">web game</a> inspired by this idea.</p><h3>Understand your goals</h3><p>How do we address some of the plausible risks above? A complete answer is another full post (or book, or lifetime). But let’s mention one piece: <strong>understand the goals you’ve given your AI</strong>. Since all AI is simply optimizing a well-defined mathematical function, that is the language you use to say what problem you want to solve.</p><p>Does that mean you should start reading up on integrals and gradient descent algorithms? I can feel your eyeballs closing. Not necessarily!</p><p>The goals are a negotiation between what your business needs (human language) and how it can be measured and optimized (AI language). You need people to speak to both sides. That is often a business or product owner in collaboration with a data scientist or quantitative researcher.</p><p>Let me give an example. Suppose you want to recommend content using a model. You choose to optimize the model to increase engagement with the content, as measured by clicks. Voila, now you understand one reason the Internet is full of clickbait: the goal is wrong. You actually care about more than clicks. Companies modify the goal without touching the AI by trying to filter out content that doesn’t meet policies. That is one reasonable strategy. Another strategy might be to add a heavy penalty to the training dataset if the AI recommends content later found to be against policy. Now we are starting to really think through how our goal affects the AI.</p><p>This example also explains why content systems can be so jumpy: you click on a video on YouTube, or a pin on Pinterest, or a book on Amazon, and the system immediately recommends a big pile of things that are almost exactly the same. Why? The click is usually measured in the short-term, so the system optimizes for short-term engagement. This is a well-known recommender challenge, centered around mathematically defining a good goal. Perhaps a part of the goal should be whether the recommendation is irritating, or whether there is long-term engagement.</p><p>Another example: if your model is accurate, but your dataset or measurements don’t look at under-represented minorities in your business, you may be performing poorly for them. Your goal may really to be accurate for all sorts of different people.</p><p>A third example: if your model is accurate, but you don’t understand why, that might be a risk for some critical applications, like healthcare or finance. If you have to understand why, you might need to use a human-understandable (“white box”) model, or explanation technology for the model you have. Understandability can be a goal.</p><h3>Conclusion: we need to understand AI</h3><p><strong>AI cannot fully replace humans</strong>, despite what you read in the popular press. The biggest difference between human and artificial intelligence is that <strong>only humans choose goals</strong>. So far, AIs do not.</p><p>If you can take away one thing about artificial intelligence: <strong>understand its goals</strong>. Any AI technology has a perfectly well-defined goal, often a labeled dataset. To the extent the definition (or dataset) is flawed, so too will be the results.</p><p>One way to understand your AI better is to explain your models. We formed Fiddler Labs to help. Feel free to reach us at <a href="/service/mailto:info@fiddler.ai">info@fiddler.ai</a>.</p><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/05/humans-choose-ai-does-not/"><em>https://blog.fiddler.ai</em></a><em> on May 8, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6512d49f7c46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/humans-choose-ai-does-not-6512d49f7c46">Humans choose, AI does not</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A gentle introduction to algorithmic fairness]]></title>
            <link>https://medium.com/data-science/a-gentle-introduction-to-algorithmic-fairness-46c825bdaf5d?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/46c825bdaf5d</guid>
            <category><![CDATA[algorithmic-fairness]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[algorithmic-bias]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 19:26:35 GMT</pubDate>
            <atom:updated>2019-08-06T21:35:16.467Z</atom:updated>
            <content:encoded><![CDATA[<h4>A gentle introduction to issues of algorithmic fairness: some U.S. history, legal motivations, and four definitions with counterarguments.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*A3MIm5vz1cLohQwj" /></figure><h3>History</h3><p>In the United States, there is a long history of fairness issues in lending.</p><p>For example, <a href="https://en.wikipedia.org/wiki/Redlining">redlining</a>:</p><p><em>‘In 1935, the Federal Home Loan Bank Board asked the Home Owners’ Loan Corporation to look at 239 cities and create “residential security maps” to indicate the level of security for real-estate investments in each surveyed city. On the maps, “Type D” neighborhoods were outlined in red and were considered the most risky for mortgage support..</em></p><p><em>‘In the 1960s, sociologist John McKnight coined the term “redlining” to describe the discriminatory practice of fencing off areas where banks would avoid investments based on community demographics. During the heyday of redlining, the areas most frequently discriminated against were black inner city neighborhoods…’</em></p><p>Redlining is clearly unfair, since the decision to invest was not based on an individual homeowner’s ability to repay the loan, but rather on location; and that basis systematically denied loans to one racial group, black people. In fact, <a href="http://powerreporting.com/color/1a.html">part 1</a> of a Pulitzer Prize-winning <a href="http://powerreporting.com/color/">series</a> in the Atlanta Journal-Constitution in 1988 suggests that location was more important than income: “ <em>Among stable neighborhoods of the same income [in metro Atlanta], white neighborhoods always received the most bank loans per 1,000 single-family homes. Integrated neighborhoods always received fewer. Black neighborhoods — including the mayor’s neighborhood — always received the fewest.</em> “</p><p>Legislation such as the 1968 <a href="https://en.wikipedia.org/wiki/Fair_Housing_Act">Fair Housing Act</a> and the 1977 <a href="https://en.wikipedia.org/wiki/Community_Reinvestment_Act">Community Reinvestment Act</a> were passed to combat these sorts of unfair practices in housing and lending.</p><p>More recently, in 2018, WUNC <a href="https://www.wunc.org/post/blacks-and-latinos-denied-mortgages-rates-double-whites">reported</a> that blacks and latinos in some cities in North Carolina were denied mortgages at higher rates than whites:</p><p><em>“Lenders and their trade organizations do not dispute the fact that they turn away people of color at rates far greater than whites. But they maintain that the disparity can be explained by two factors the industry has fought to keep hidden: the prospective borrowers’ credit history and overall debt-to-income ratio. They singled out the three-digit credit score — which banks use to determine whether a borrower is likely to repay a loan — as especially important in lending decisions.”</em></p><p>The WUNC example raises an interesting point: it is possible to look unfair via one measure (loan rates by demographic), but not by another (ability to pay as judged by credit history and debt-to-income ratio). Measuring fairness is complicated. In this case, we can’t tell if the lending practices are fair because the data on credit history and debt-to-income ratio for these particular groups are not available to us to evaluate lenders’ explanations of the disparity.</p><p>In 2007, the federal reserve board (FRB) <a href="https://www.federalreserve.gov/boarddocs/rptcongress/creditscore/">reported</a> on credit scoring and its effects on the availability and affordability of credit. They <a href="https://www.federalreserve.gov/boarddocs/rptcongress/creditscore/executivesummary.htm">concluded</a> that the credit characteristics included in credit history scoring models are not a proxy for race, although different demographic groups have substantially different credit scores on average, and <em>“for given credit scores, credit outcomes — including measures of loan performance, availability, and affordability — differ for different demographic groups.”</em> This FRB study supports the lenders’ claims that credit score might explain disparity in mortgage denial rates (since demographic groups have different credit scores), while also pointing out that credit outcomes are different for different groups.</p><p>Is this fair or not?</p><h3>Defining fairness</h3><p>As machine learning (ML) becomes widespread, there is growing interest in fairness, accountability, and transparency in ML (e.g., the <a href="https://fatconference.org/">fat*</a> conference and <a href="http://www.fatml.org/">fatml</a> workshops).</p><p>Some researchers say that fairness is not a statistical concept, and no statistic will fully capture it. There are many statistical definitions that people try to relate to (if not define) fairness.</p><p>First, here are two legal concepts that come up in many discussions on fairness:</p><ol><li><a href="https://en.wikipedia.org/wiki/Disparate_treatment"><strong>Disparate treatment</strong></a>: “unequal behavior toward someone because of a protected characteristic (e.g., race or gender) under Title VII of the United States Civil Rights Act.” Redlining is disparate treatment if the intent is to deny black people loans.</li><li><a href="https://en.wikipedia.org/wiki/Disparate_impact"><strong>Disparate impact</strong></a>: “practices .. that adversely affect one group of people of a protected characteristic more than another, even though rules applied .. are formally neutral.” (<em>“The disparate impact doctrine was formalized in the landmark U.S. Supreme Court case Griggs v. Duke Power Co. (1971). In 1955, the Duke Power Company instituted a policy that mandated employees have a high school diploma to be considered for promotion, which had the effect of drastically limiting the eligibility of black employees. The Court found that this requirement had little relation to job performance, and thus deemed it to have an unjustified — and illegal — disparate impact.”</em> [Corb2018])</li></ol><p>[Lipt2017] points out that these are legal concepts of disparity, and creates corresponding terms for technical concepts of parity applied to machine learning classifiers:</p><ol><li><strong>Treatment parity</strong>: a classifier should be blind to a given protected characteristic. Also called anti-classification in [Corb2018], or “fairness through unawareness.”</li><li><strong>Impact parity</strong>: the fraction of people given a positive decision should be equal across different groups. This is also called demographic parity, statistical parity, or independence of the protected class and the score [Fair2018].</li></ol><p>There is a large body of literature on algorithmic fairness. From [Corb2018], two more definitions:</p><ol><li><strong>Classification parity</strong>: some given measure of classification error is equal across groups defined by the protected attributes. [Hard2016] called this <em>equal opportunity</em> if the measure is true positive rates, and <em>equalized odds</em> if there were two equalized measures, true positive rates and false positive rates.</li><li><strong>Calibration</strong>: outcomes are independent of protected attributes conditional on risk score. That is, reality conforms to risk score. For example, about 20% of all loans predicted to have a 20% chance of default actually do.</li></ol><p>There is lack of consensus in the research community on an ideal statistical definition of fairness. In fact, there are impossibility results on achieving multiple fairness notions simultaneously ([Klei2016] [Chou2017]). As we noted previously, some researchers say that fairness is not a statistical concept.</p><h3>No definition is perfect</h3><p>Each statistical definition described above has counterarguments.</p><p>Treatment parity unfairly ignores real differences. [Corb2018] describes the case of the COMPAS score used to predict recidivism (whether someone will commit a crime if released from jail). After controlling for COMPAS score and other factors, women are less likely to recidivate. Thus, ignoring sex in this prediction might unfairly punish women. Note that the <a href="https://www.consumer.ftc.gov/articles/0347-your-equal-credit-opportunity-rights">Equal Credit Opportunity Act</a> legally mandates treatment parity: “ <em>Creditors may ask you for [protected class information like race] in certain situations, but they may not use it when deciding whether to give you credit or when setting the terms of your credit.</em> “ Thus, [Corb2018] implies that this sort of unfairness is enshrined in law.</p><p>Impact parity doesn’t ensure fairness (people argue against quotas), and can cripple a model’s accuracy, harming the model’s utility to society. [Hard2016] discusses this issue (using the term “demographic parity”) in its introduction.</p><p>Corbett et al. [Corb2018] argue at length that classification parity is naturally violated: “ <em>when base rates of violent recidivism differ across groups, the true risk distributions will necessarily differ as well — and this difference persists regardless of which features are used in the prediction.</em> “</p><p>They also argue that calibration is not sufficient to prevent unfairness. Their hypothetical example is a bank that gives loans based solely on the default rate within a zip code, ignoring other attributes like income. Suppose that (1) within zip code, white and black applicants have similar default rates; and (2) black applicants live in zip codes with relatively high default rates. Then the bank’s plan would unfairly punish creditworthy black applicants, but still be calibrated.</p><h3>Conclusion</h3><p>In summary, likely fairness has no single measure. We took a whirlwind tour of four statistical definitions, two motivated by history and two more recently motivated by machine learning, and summarized the counterarguments to each.</p><p>This also means it is challenging to automatically decide if an algorithm is fair. Open-source fairness-measuring packages reflect this by offering many different measures.</p><p>However, this doesn’t mean we should ignore statistical measures. They can give us an idea of whether we should look more carefully. Food for thought. We should feed our brain well, it being the most likely to make the final call.</p><p>(A note: this subject is rightfully contentious. Our intention is to add to the conversation in a productive, respectful way. We welcome feedback of any kind.)</p><p><em>Thanks to </em><a href="https://twitter.com/kkenthapadi"><em>Krishnaram Kenthapadi</em></a><em>, </em><a href="https://twitter.com/zacharylipton"><em>Zack Lipton</em></a><em>, </em><a href="https://twitter.com/lukemerrick_"><em>Luke Merrick</em></a><em>, </em><a href="https://twitter.com/amitpaka"><em>Amit Paka</em></a><em>, and </em><a href="https://twitter.com/krishnagade"><em>Krishna Gade</em></a><em> for their feedback.</em></p><h3>References</h3><ul><li>[Chou2017] Chouldechova, Alexandra. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.” Big Data 5, no. 2 (June 1, 2017): 153–63. <a href="https://doi.org/10.1089/big.2016.0047">https://doi.org/10.1089/big.2016.0047</a>.</li><li>[Corb2018] Corbett-Davies, Sam, and Sharad Goel. “The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning.” ArXiv:1808.00023 [Cs], July 31, 2018. <a href="http://arxiv.org/abs/1808.00023">http://arxiv.org/abs/1808.00023</a>.</li><li>[Fair2018] “Fairness and Machine Learning.” Accessed April 9, 2019. <a href="https://fairmlbook.org/">https://fairmlbook.org/</a>.</li><li>[Hard2016] Hardt, Moritz, Eric Price, and Nathan Srebro. “Equality of Opportunity in Supervised Learning.” ArXiv:1610.02413 [Cs], October 7, 2016. <a href="http://arxiv.org/abs/1610.02413">http://arxiv.org/abs/1610.02413</a>.</li><li>[Klei2016] Kleinberg, Jon, Sendhil Mullainathan, and Manish Raghavan. “Inherent Trade-Offs in the Fair Determination of Risk Scores.” ArXiv:1609.05807 [Cs, Stat], September 19, 2016. <a href="http://arxiv.org/abs/1609.05807">http://arxiv.org/abs/1609.05807</a>.</li><li>[Lipt2017] Lipton, Zachary C., Alexandra Chouldechova, and Julian McAuley. “Does Mitigating ML’s Impact Disparity Require Treatment Disparity?” ArXiv:1711.07076 [Cs, Stat], November 19, 2017. <a href="http://arxiv.org/abs/1711.07076">http://arxiv.org/abs/1711.07076</a>.</li></ul><p><em>Originally published at </em><a href="https://blog.fiddler.ai/2019/04/a-gentle-introduction-to-algorithmic-fairness/"><em>https://blog.fiddler.ai</em></a><em> on April 23, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46c825bdaf5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-gentle-introduction-to-algorithmic-fairness-46c825bdaf5d">A gentle introduction to algorithmic fairness</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case study: explaining credit modeling predictions with SHAP]]></title>
            <link>https://medium.com/fiddlerlabs/case-study-explaining-credit-modeling-predictions-with-shap-2a7b3f86ec12?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a7b3f86ec12</guid>
            <category><![CDATA[peer-to-peer-lending]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[shap]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Thu, 21 Mar 2019 23:21:51 GMT</pubDate>
            <atom:updated>2019-03-21T23:23:25.959Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>At Fiddler labs, we are all about explaining machine learning models. One recent interesting explanation technology is SHAP (SHapely Additive exPlanations). To learn more about how SHAP works in practice, we applied it to predicting loan defaults in data from Lending Club.</p><p>We built three models (random, logistic regression, and boosted trees), and looked at several feature explanation diagrams from each.</p><p>We are not financial experts, so this blog post focuses on comparing algorithms, not insights into loan data.</p><p>Hopefully, we can bring SHAP into sharper focus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TUJ82Tg2RJCaG-Hv" /></figure><h3>What is SHAP?</h3><p>SHAP (SHapely Additive exPlanations) is a way to build a linear explanation model of feature importance for a given prediction. It works for any model, even non-linear ones like boosted trees or deep learning neural networks. Some model types (like logistic regression or trees) have a fast exact version, while for others there are tradeoffs between approximations, speed, and sampling variability.</p><p>SHAP is appealing because it is based on the Shapley values from game theory, which are the only explanations within a broad class of possibilities that satisfy three useful properties, termed by Lundberg et al.:</p><p>1. local accuracy (roughly: the explanation model matches the real model at the point being explained)</p><p>2. missingness (Lundberg: “missingness requires features missing in the original input to have no impact”)</p><p>3. consistency (Lundberg: “Whenever we change a model such that it relies more on a feature, then the attributed importance for that feature should not decrease.”)</p><p>Lundberg <a href="https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27">says</a> consistency allows us to meaningfully compare the feature importance between two models.</p><p>We won’t further re-explain SHAP here, since that would be a post by itself. For that, see <a href="https://github.com/slundberg/shap">the shap library</a>, a <a href="https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27">blog post by the author</a>, the <a href="https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf">2017 NIPS paper</a>, and a <a href="https://towardsdatascience.com/one-feature-attribution-method-to-supposedly-rule-them-all-shapley-values-f3e04534983d">nice blog post by Cody Marie Wild</a>. Note the <a href="https://github.com/slundberg/shap">SHAP python library</a> (link) generated many of the graphs in this blog post.</p><p>Also, we’ve integrated SHAP into the <a href="http://fiddler.ai/">Fiddler Explainable AI Engine</a>.</p><h3>The data</h3><p>We took a set of loans at least 3 years old that were either paid (“Fully Paid”) or in default (“Charged Off”) from Lending Club data. You can download our cleaned dataset from github (<a href="https://github.com/fiddler-labs/p2p-lending-data">link</a>). If you wish to look at the meaning of variable names, we used mostly a subset of the variables from this Kaggle dataset (<a href="https://www.kaggle.com/wordsforthewise/lending-club">link</a>) with this data dictionary (<a href="https://resources.lendingclub.com/LCDataDictionary.xlsx">link</a>).</p><p>We also cleaned the data in a variety of ways detailed in <a href="https://gist.github.com/dfrankow/9e42b4c777d8230d42538f9a494c14cc">this Jupyter notebook</a> (for example, removing variables with missing values). Also, we removed variables not available at loan application time (e.g., total_pymnt, the total amount paid), variables from Lending Club’s own model (e.g., int_rate, the interest rate, or sub_grade, the loan grade), and issue date (which might encode general economic conditions).</p><p>We chose the first 294,520 records (in date order) as a model training set, the next 56,568 as a validation set, and the final 119,015 rows as a test set. We train on the training set, and do our analysis on the validation set. (We never use the test set in this post.)</p><p>The default rate (of “Charging Off”) is 13% in training, and 15% in the validation set.</p><h3>The models</h3><p>We built models to predict whether a loan was “Charged Off”, i.e. not paid in full.</p><p>We used scikit-learn 0.20.2 to run a random predictor and a logistic regression (the old linear workhorse), lightGBM 2.2.3 for boosted decision trees, and SHAP library 0.28.5.</p><p>For modeling, we took a “kitchen sink” approach (i.e., toss in most of the variables), because people are often tempted to “let the machine sort it out.” (In practice, this may not be best. Choosing variables may depend on how much data you have, what it means, what your purpose for the model is, how well your model technology can handle highly related variables, and so on.) Our models have 130 features after pre-processing.</p><p>AUC (area under an ROC curve, a common measure of prediction accuracy) was 0.5 for random (as we’d expect), and 0.68 for both logistic and lightGBM. These are not fantastically accurate models, but better than random, which is what we need for learning purposes. Also, if we could predict loan default perfectly, we’d design faster-than-light travel next!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LL3r5gYP9f8cUx89.jpg" /><figcaption>Speeding into the future after perfectly predicting loan default.</figcaption></figure><h3>Measuring SHAP values on the models</h3><p>We got SHAP explanatory values on a subsample of the validation data for each model. Let’s walk through what that means.</p><p>We chose a 1,000-row subsample from our validation set, to speed up the time-consuming SHAP calculation.</p><p>We used shap.KernelExplainer, which uses sampling with a kernel to estimate the SHAP values, as described in <a href="https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf">the SHAP paper</a>. Roughly: for each feature in a single row of validation data, it changes the other features to randomly chosen values from other data points (that SHAP calls the <em>background data</em>), weighted by a kernel function to preserve the SHAP properties. That produces a large set of data to which it fits a linear model. It is a “local” model because the linear explanation model is fit around this single row of original data.</p><p>There are other, more specific explainers for our model types: we could use shap.LinearExplainer for our logistic model, and shap.TreeExplainer for our boosted trees model. They would be faster, and produce exact values. However, for simplicity we’ll use KernelExplainer for everything. (We actually checked exact versions with slightly different parameters, and they look similar, though not the same. We might post again on this topic.)</p><p>For background data, it is slow to use the whole training dataset, so (as recommended by the library) we used shap.kmeans to create 30 representative rows from another subsample of 20,000 points from the training set.</p><p>We run that SHAP local model for each of the 1,000 rows, so we end up with 1,000 SHAP values for each feature.</p><p>Why 20,000? Why 30? Why 1,000? This is all trading off amount of data for speed, to try to estimate SHAP values in minutes instead of hours or days.</p><p>Moreover, if you look at the shap code carefully, there are a number of other choices buried in the code. For example, the l1_reg parameter of the <a href="https://github.com/slundberg/shap/blob/master/shap/explainers/kernel.py#L132">shap_values</a> function says auto (the default) ‘uses “aic” [to regularize feature selection] when less that 20% of the possible sample space is enumerated, otherwise it uses no regularization.’ We ended up changing the l1_reg parameter to a different version of regularization to choose 20 features, because that is what we would use for this blog post.</p><p>There are many nuances to understand, or choices to make, or perhaps defaults to accept. Such is the life of a data scientist or machine learning engineer: it’s messy putting theory into practice.</p><h3>Random classifier</h3><p>First, as a sanity check, we tried a random classifier (DummyClassifier from scikit-learn, set to the default settings including the ‘stratified’ strategy, meaning it predicts randomly with the distribution of the training data).</p><p>Below is a histogram of the predicted default probabilities, showing mostly zeros (in line with the rate of default in the training set):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*RLAyfmZKQ30H65Ht" /></figure><p>Below is an ROC curve, showing that the result are completely random (right on the diagonal). We don’t really need a graph, we could just report AUC of 0.5. However, the graph really brings it home.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/0*kJ1TmRGy5x5wgOKJ" /></figure><p>Below are the mean absolute SHAP values for the 1,000 rows of our validation subsample from KernelExplainer. These are indicators of how important each feature is to prediction. The default graph shows the top 20 features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bvorWhTDtIPGjtKq" /></figure><p>There is some variation, but as we’d expect for a random predictor, most features are of similar importance. Also, we’ll see later that these mean values are small compared to the other models.</p><p>This is what the “missingness” property of SHAP looks like in random data. Really, none of these features are important. We could hope they would show up with zero importance, but apparently that would be too much to hope for. Since this is an average of absolute values of a bunch of random things, what we get is “small and undifferentiated importance”, as long as we have enough samples.</p><p>Below is a graph of every SHAP value for every feature in each of the 1,000 rows of the validation subsample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7viGJaYdJ_gBCgeG" /></figure><p>Here’s how you interpret this graph:</p><ul><li>as the feature value goes higher, the color moves from blue to red</li><li>the X axis shows the impact on a model prediction</li><li>the Y axis shows the feature, sorted by mean absolute SHAP value</li></ul><p>These are the same 20 features as the previous graph, because this graph also sorts features by mean absolute SHAP value.</p><p>The graph looks to have fairly similar shapes for all the features. The tail is longer and thinner to the right than to the left, which is not an obvious outcome. It might have to do with the overall rate of default. (Note there is a visual gap in feature importances between 0 and 0.01. That is because of the regularization l1_reg parameter in shap_values that explicitly chooses the top 20 features. If we change that parameter, those gaps vanish.)</p><p>Red and blue intermingle freely, because feature values are not related to model predictions. So for any particular feature value (color), there will be some data rows with high SHAP values, and some with low SHAP values.</p><h3>Logistic regression</h3><p>Next we tried a logistic regression, a call to <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">LogisticRegression</a> that used all default values in scikit-learn 0.20.2. This is a logistic regression with some L2 regularization.</p><p>Below is a histogram of the predicted default probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*sw72zpOUMOmwo7MW" /></figure><p>They are roughly centered around the background rate of default in the training set (13%), but occasionally the model is slightly more certain of default.</p><p>Below is an ROC curve with an AUC of 0.68, showing that the results are not completely random (since they are above the diagonal).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*nRzQk_nAHxqsRzCW" /></figure><p>For logistic regression, the SHAP overall (mean absolute) feature impact should be similar to the logistic regression coefficients.</p><p>Another way to think of this is that “local” explanations (SHAP linear approximations of model predictions around each data point) and “global” explanations (model coefficients) should be similar, although some factors will prevent an exact match, e.g., l1 regularization of the features (see the notebook).</p><p>Let’s look at regression coefficients versus SHAP values. Below are the regression coefficients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/0*IPhb2-CEjQZXOXGL" /></figure><p>Below are the SHAP importance values. We use the link=’logit’ parameter of KernelExplainer to make the feature values log-odds units, hence more comparable to the logistic regression. (An identity link function might be more human-interpretable because it is in probability units, but would likely match the logistic coefficients more poorly.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8eP_r_Og1reDEm9F" /></figure><p>First, the explanations agree a lot: 15 of the top 20 variables are in common between the top logistic regression coefficients and the SHAP features with highest mean absolute SHAP values. Below is a graph comparing these values for the top 20 logistic coefficients:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*1529Ty9SYtERWPKM" /></figure><p>Second, the values are often similar in magnitude, though not the same. For example, the absolute value of the total_bc_limit coefficient is about 0.25 in the logistic fit, and the SHAP mean absolute importance value looks to be about 0.18.</p><p>In fact, the SHAP importance values all seem a bit lower than the absolute values of the logistic coefficients. Perhaps the regularized logistic regression and SHAP treat <a href="https://en.wikipedia.org/wiki/Multicollinearity">collinear</a> (highly related) features slightly differently. Or perhaps the multiple levels of sampling in SHAP (both in the subsample we chose, and in the feature values chosen per data row) shift things around a bit.</p><p>We looked at a subsample of size 2,000 instead of 1,000, to see if the SHAP values would shift a lot. Below are a graph of those values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L85JQHrDINgXNzQe" /></figure><p>They don’t shift much. Between a subsample of 1,000 and 2,000, all 20 variables are the same, in almost the same order, with almost the same magnitude. That gives us confidence that these values are stable.</p><p>Now let’s look at a more detailed display that shows SHAP values for the 1,000 points in our subsample of the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mF5D4T0jdrmeKUJl" /></figure><p>Let’s look at the top feature: total_acc. If the feature value is low (blue), the model prediction (probability of default) tends to be higher (X value is positive). I don’t know this data well enough to understand why having more accounts would make you less likely to default. Perhaps people who are trying to consolidate their debt have a lot of accounts at application time, and those people are more likely to pay off their consolidation loan? Or, perhaps this feature is highly correlated to another feature that predicts risk and the model has decided to use this feature to cancel out the effects of that other feature. The open_acc feature is similar.</p><p>The feature ‘fico_range_midpoint’ makes more sense to me. It represents the applicant’s credit score at time of application. If credit score is higher (red), predicted probability of default goes down (X value is negative).</p><p>Several other observations:</p><p>1. Looking at data instead of averages shows that some features (e.g., total_bc_limit with its long tail) have more outliers in SHAP values than other features (e.g., dti, with its squat shape).</p><p>2. SHAP effects are monotonic for a linear model: the color always goes smoothly from one color to the other across the graph. (Logistic regression is linear enough.)</p><p>3. Integer-valued features (like acc_open_past_24_mths or delinq_2yrs) map to evenly spaced SHAP values for a linear model</p><p>4. We can see indicator variables (e.g., “emp_length_10+ years”). Features with a smaller SHAP value impact (e.g., “emp_length_10+ years”) may have a higher overall (mean absolute) impact if they occur more often, compared to a larger impact but more rare (like purpose_credit_card).</p><p>5. Many features look to have small impact, but with a long tail.</p><h3>lightGBM (boosted trees)</h3><p>Now let’s look at boosted trees. We make a plain vanilla default call to lightgbm.train in <a href="https://lightgbm.readthedocs.io/en/latest/Features.html">lightGBM</a> 2.2.3. Now there is no easily summarized global explanation like “coefficients.”</p><p>Below is a graph of the predicted default probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/0*SdHcTD1eJrw98vXk" /></figure><p>As with the logistic model, these values are roughly centered around the probability of default in the training data.</p><p>Below is an ROC curve with an AUC of 0.68, showing that the results are not completely random (above the diagonal). We didn’t gain much over logistic in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*6mdGiSUmhy89Jsfs" /></figure><p>Below is a list of lightGBM feature importances, as reported by the <a href="https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.Booster.feature_importance">feature_importance function</a>, with importance_type=’gain’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*mYoc2oMIQXRQvu-I" /></figure><p>And below is a graph of SHAP feature importances (mean absolute SHAP value).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7BBHlk6zSvijrOLj" /></figure><p>Lundberg (the inventor of SHAP) explains that SHAP values disagree with information gain in <a href="https://towardsdatascience.com/interpretable-machine-learning-with-xgboost-9ec80d148d27">his blog post about XGBoost</a>, and why he trusts SHAP. In short, information gain attributes importance inconsistently when Lundberg makes a small change to a toy model (making “cough” +10 in the red model), because information gain gives more weight to nodes lower in the tree, when it should do the opposite (root nodes are more important).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*CojOH31yX9Qx_rAt" /></figure><p>After all that discussion of why they might be different, these SHAP mean absolute values agree surprisingly well with the lightGBM information gain. Below is a chart where we scale SHAP values to be visually comparable to the information gain values. The relative values of the features look quite similar. (Note: we have seen other data where this is not true.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/0*4s27uUPGmp_bh8nR" /></figure><p>Below is a graph of all the SHAP values of a 1,000-row subsample of the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WhGBe_ZDbVq0-9k-" /></figure><p>First, not all the features are monotonic. That is, the color doesn’t slide uniformly from red to blue. For example, if you look closely, the top feature annual_inc is a mixed cloud of blue and red for negative values. If I were working on this problem for real, I’d want to dig into that. It could be that there are different types of loan applications, for example dividing into quadrants: feature low/SHAP low, feature low/SHAP high, feature high/SHAP low, feature high/SHAP high.</p><p>However, many features are still monotonic. For example, high fico_range_midpoint (red) means a lower probability of default (X axis negative). Or dti (debt-to-income ratio), a feature people expect to be monotonically correlated to default rate, and it is (mostly).</p><p>(Side note: the average absolute SHAP values for boosted trees are an order of magnitude lower than those for logistic regression. This is because we ran logistic SHAP with a logit link function in order to make the SHAP values comparable to the logistic regression, but we ran the boosted trees SHAP without a logit link function.)</p><p>Below is the same graph, but on a 2,000-row subsample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LiLmagULDO9kyh5a" /></figure><p>It is very similar-looking to the 1,000-row subsample. All top 20 variables are the same (with a couple of swaps in order). All the graphs look substantially graphically similar. This again gives us confidence that these measurements are stable.</p><h3>Comparing logistic regression and boosted trees</h3><p>Logistic regression and boosted trees had similar accuracy, an AUC of 0.68. Their predicted values also agree (correlation coefficient of 0.86).</p><p>However, their SHAP values don’t agree, likely because we used a logit link function for logistic but not boosted trees.</p><h3>Summary</h3><p>We looked at explanations for three models to predict defaults in loan data.</p><p>In the two models that are easier to globally understand, SHAP mostly matched our expectations.</p><ul><li>Explanations for a random model show that no features stand out. All features contribute similar small amounts to the predictions.</li><li>Explanations for a logistic regression model show features that mostly correspond to the global regression coefficients.</li></ul><p>In the case of boosted trees (a non-linear model), we had fewer expectations about what SHAP would say about the features. As it turned out, the logistic and boosted trees model do agree on feature importance, but not SHAP values, because we used different link functions for logistic regression and boosted trees.</p><p>Also, the detailed dataset SHAP values diagram added more to our understanding of each feature: we could see which features were monotonic and which were not; we could see long tails and the relative importance and strength of indicator variables; and so on.</p><p>Finally, we looked at one way to investigate the stability of SHAP by eye: compute a larger sample, and see if SHAP feature summary values look stable.</p><p>In conclusion: SHAP helps explain the features most important to model predictions, but there is still a lot of nuance to understanding and using it correctly.</p><p><em>Thanks to </em><a href="https://medium.com/@peteskomoroch"><em>Peter Skomoroch</em></a><em>, </em><a href="https://medium.com/u/54932a03b331"><em>Luke Merrick</em></a><em> and </em><a href="https://medium.com/@scottmlundberg"><em>Scott Lundberg</em></a><em> for their feedback.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a7b3f86ec12" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fiddlerlabs/case-study-explaining-credit-modeling-predictions-with-shap-2a7b3f86ec12">Case study: explaining credit modeling predictions with SHAP</a> was originally published in <a href="https://medium.com/fiddlerlabs">fiddlerlabs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mary and John: using first name to predict sex in the US works quite well]]></title>
            <link>https://medium.com/@dfrankow/using-first-name-to-predict-sex-in-the-us-23b6028cff68?source=rss-abc2df62e05b------2</link>
            <guid isPermaLink="false">https://medium.com/p/23b6028cff68</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gender]]></category>
            <dc:creator><![CDATA[Dan Frankowski]]></dc:creator>
            <pubDate>Mon, 11 Feb 2019 16:49:57 GMT</pubDate>
            <atom:updated>2019-05-01T01:39:54.519Z</atom:updated>
            <content:encoded><![CDATA[<p>Someone’s first name is a good clue of their sex. Mary is probably female. John is probably male. How good a clue exactly? I use United States Social Security Administration (SSA) birth records to take a look. The short story: <strong>in the US, first name is a very good clue of sex at birth</strong>.</p><p>I use <a href="https://www.ssa.gov/OACT/babynames/limits.html">SSA US “national” data</a>, which has name, sex, and number of births for the years 1880–2017 for the entire country; 348,120,517 births in total.</p><p>This data only has binary sex (F or M). I use the word “sex” deliberately. “Gender” refers to a cultural identity, while “sex” <a href="https://en.wikipedia.org/wiki/Sex_and_gender_distinction">refers</a> to biological characteristics, and more to the point, what someone put on an application for a social security number. I don’t want to exclude people with non-binary gender identity, but sex at birth is tracked in a binary way by the SSA, which is the data I could think of to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RvmdCcsKXzis2_4lXWuBHw.png" /></figure><h3>Results</h3><p>Suppose we assume the more common sex for a name is always true. An example: the name Pat has 66,854 births in this dataset, of which 40,123 are female. Suppose we say Pat is always female. We’re right about 60% of the time. Not very accurate.</p><p>However, if we assume the more common sex for a name is always true, that is correct for 98% of births in the dataset. Near perfect for social data. If we look at the accuracy of that label by year, it trends down, but stays over 95%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6R2DmcVjr8nv5Kr6hnefyQ.png" /></figure><p>Our previous example (Pat) shows that some names are more mixed-sex than others. Below is a graph by birth year for fraction of “Pat” births who were female, with only the years with at least 100 Pats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-k46WSyH_IBdCMn15y16A.png" /></figure><p>Pretty fun. It started out more male (only 30% female in the 1910s and 1920s), went almost all female (up to 80% in 1940), then has veered back towards male. The last point is 1973, after which “Pat” has fewer than 100 births in a year. Note this is only exactly “Pat”, it doesn’t include “Patrick” or “Patricia”.</p><p>Below are the top 30 most mixed-sex names by year, with fraction of births that are female. A name near 100% is mostly female, and near 0% is mostly male. There are only dots for at least 100 births in that year. Also, it only shows data since 1920 because there are more interesting movements there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdXeeehqlspZjZdbT79bBw.png" /></figure><p>Examples:</p><ul><li>Alva was always in the middle, and became rare after 1964</li><li>Amari became popular in 1994</li><li>“Baby” and “Unknown” are likely not real names. We’d expect them to be around 50%, and they are, mostly</li><li>Some names started out more male, and became more female, such as Riley, Stevie, Blair, Quinn, Mckinley, Emery, Ivory, Jaylin. For Stevie, a friend guessed <a href="https://en.wikipedia.org/wiki/Stevie_Nicks">Stevie Nicks</a>, who joined Fleetwood Mac in 1975. For Blair, that really turned female in 1980–1982 that same friend guessed Blair in “<a href="https://www.imdb.com/title/tt0078610/">The Facts of Life</a>”, a television show on 1979–1988.</li><li>Some names started out more female, and became more male, such as Robbie. Robbie is heading male in 1950, but moves that way faster in 1960, possibly due to Robbie in <a href="https://en.wikipedia.org/wiki/My_Three_Sons">My Three Sons</a>, yet another tv series, that started in 1960 (pointed out by yet another of my friends!).</li><li>Some have stayed in the middle, or veered back and forth, such as Frankie, Jackie, Justice, Kerry, Remy</li><li>Jaime is interesting. A friend guessed that it is a combination of a Spanish name for a boy that stuck around, and a newer variant for an English name “Jamie” that came charging in around 1975, then lost steam. Another friend guessed <a href="https://en.wikipedia.org/wiki/Jaime_Sommers_(The_Bionic_Woman)">Jaime Sommers</a>, the (fictional) bionic woman, who first appeared in “The Six Million Dollar Man”, a popular tv show, and got her own show 1976–1978.</li></ul><p>I don’t have earth-shattering conclusions. (Popular culture influences names?) But, I enjoy looking at data. It does seem maybe names go towards female more often than towards male? I could count that more precisely with more work.</p><p>One more graph: the same one, but for the names that are most mixed-sex as of 1970.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XyBG-q-9Dx1-Vdm1KLSCCQ.png" /></figure><p>I still find it fascinating that these names can have such different shapes. Riley, Raleigh, Harley, Emory, Leighton, Emerson went from male to female; Jan went from female to male (again, this might be a combination of multiple names with the same spelling), and Jody may be headed there.</p><p>In conclusion: <strong>overall, first names predict sex well, but there are also some mixed-sex names, and some that change over time</strong>.</p><h3>Discussion</h3><p>So why should you care? (The first and only question to answer for any presentation.)</p><p>First, I validated the idea, going from a guess (“first name likely tells us about sex”) to a strong indication with data (98% accuracy overall). Of such small steps is science made. Second, I described a method (grab SSA data, look at most common sex per name) that is easy to implement and easy to interpret. Third, I dug into the data to find some fun examples.</p><p>Is there danger in publishing this result? (A question every data scientist should ask.) The answer is rarely obvious. This method is a tool. It allows someone to take a dataset with first names and add gender with fairly high accuracy in the US. So, it may enable analyses that were not possible before.</p><p>Like any tool, it can be used for good or ill. It might help a researcher look at prevalence of men and women in the news to press for gender equality. Or, it might help a company spy on people. (I published related privacy risks in 2006 in “<a href="https://www.cs.cornell.edu/~danco/research/papers/privacy-sigir2006.pdf">You are what you say: privacy risks of public mentions</a>”.)</p><p>Or it might be misused. For example, this method may be less accurate for names from disadvantaged populations like women or minorities. Looking at a few examples it looks pretty accurate, but that’s not science. I have no systematic evidence.</p><p>I think this is interesting and may be a valuable tool for good. Moreover, the cat is out of the bag. If I search for “predicting gender from name” I see services, programming APIs, code repositories, and published results. It seems likely that 10 times as many people have already done this in private companies where it wouldn’t show up publicly.</p><p>For all these reasons, I’m willing to publish it.</p><h3>Next steps</h3><p>How could we make this model more accurate? We might be able to use birth year, though we might not have that information in every setting.</p><p>Better yet, we could use spelling. Michael is likely male, Michaela is likely female. The ending of the word might tell us something. Also, names like “Patrick” and “Patricia” are related: they both shorten to “Pat”. But, this requires work.</p><p>Here is perhaps the real lesson: <strong>do the simple thing</strong> (usually counting), see there is a useful result, then move forward. Perfect is the enemy of good.</p><p>Maybe I’ll do another post using this data.</p><p><em>Thanks to Melissa, John, and Cole for great feedback and name examples.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23b6028cff68" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>