<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hanne-Torill Mevik on Medium]]></title>
        <description><![CDATA[Stories by Hanne-Torill Mevik on Medium]]></description>
        <link>https://medium.com/@htmevik?source=rss-6f1e1b2254e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nwnkFOf0PlcXqHNr1bBLUg.jpeg</url>
            <title>Stories by Hanne-Torill Mevik on Medium</title>
            <link>https://medium.com/@htmevik?source=rss-6f1e1b2254e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:08:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@htmevik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Good Data Science — Interpreting the Black Box]]></title>
            <link>https://medium.com/tech-accounts/good-data-science-interpreting-the-black-box-27d1d1939b35?source=rss-6f1e1b2254e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/27d1d1939b35</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[interpretability]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <dc:creator><![CDATA[Hanne-Torill Mevik]]></dc:creator>
            <pubDate>Fri, 15 Mar 2019 11:07:42 GMT</pubDate>
            <atom:updated>2019-03-15T11:07:42.819Z</atom:updated>
            <content:encoded><![CDATA[<h3>Good Data Science — Interpreting the Black Box</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcQWuvlZ2-tafjhgskYROQ.png" /><figcaption>Illustration by <a href="https://hanne-morstad.myportfolio.com/">Hanne Morstad</a></figcaption></figure><p>In <a href="/service/ttps://medium.com/@htmevik/how-to-good-data-science-democratized-ai-d70145e4a606">the previous</a> article in this series about Democratized AI and the Black Box Problem, I proposed a checklist that we should be able to answer with some degree of confidence in order to assess if one has achieved Interpretability within a Machine Learning system. That is, given our hypothesis and the questions we are trying to answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOcdhvsS3yaNhmZVyfkRnw.png" /></figure><p>In this post I will provide examples and attempt to argue the relevance of these criteria and how they can guide us towards doing Good Data Science.</p><h3>What are you looking at?</h3><p>Neural Networks, and especially Deep Neural Networks, outperforms almost any other Machine Learning techniques for a number of tasks, because they are able to learn <strong><em>machine learning models</em></strong> that captures the high-dimensional, non-linear correlations contained in the data, represented in a parameter space with millions of parameters. In practice, decoupling and understanding these learned correlations is an intractable task. But we can to some extent <em>visualize</em> these correlations and gain some intuition about the inner workings of the network.</p><p>There are currently two main approaches to inspect <em>how and what </em>a Neural Network has learned: <strong>Layer attribution </strong>and <strong>Feature visualization</strong>. The latter will be discussed in a follow-up post going deeper into Interpretability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/0*iuFsvDNOc1CBBHam.png" /><figcaption>A two-class problem, retriever puppy and kitten, saliency map on the right. Credit: “Feature Visualization” by Olah, et.al., Distill, 2017.</figcaption></figure><p><strong>Layer attribution</strong>, also called a <em>saliency map</em>, measures which parts of an image were the most important contributions in having a model predict a particular object class. Shown in the figure, a classifier model has determined that there are two classes in the image: a retriever puppy and a kitten. The heat map indicates which <a href="https://en.wikipedia.org/wiki/Saliency_map">super pixels</a> belong to which class, and also the relative contribution, or importance, of each pixel in determining each objects class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/0*BZ-7VVQ116hst5W5.png" /><figcaption>A sailboat and terrier, with their respective saliency maps below. K. Simonyan et.al. arXiv:1312.6034v2</figcaption></figure><p>Similarly, in the next figure, we can see that the top of the sail was very influential in the classification of “sail boat”, with some contribution also by the boat being surrounded by a specific texture, which we can identify as choppy sea. The example to the right with the terrier, is actually quite impressive, because even when the dog is imaged with a background of similar colour tones, the model has learned that the<em> colour itself is not an important feature</em> to look for in determining the object class. (<a href="https://medium.com/bakken-b%C3%A6ck/how-biased-ai-can-teach-us-to-be-better-humans-3be60ebbe558">This article provides</a> examples of how colour as an important feature can lead to racist and discriminatory Machine Learning systems)</p><h3>Q: Is the model answering the right question?</h3><p>A practical use case for Layer Attribution is to check whether or not the classification model is <strong>answering</strong> <strong>the right question</strong>. A famous example is the <a href="http://innovation.uci.edu/2017/08/husky-or-wolf-using-a-black-box-learning-model-to-avoid-adoption-errors/">Wolf v Husky</a> classifier, where a classifier model has been trained to answer the question of whether the animal in the images is a husky or a wolf. And seemingly, the performance is quite good and the model gets most classes right. That is, until one inspects <strong><em>where in the image</em></strong> the model is actually looking. It turns out that during prediction the model is actually looking at the background and not paying much attention to the animal itself. In other words, the model is actually answering the question “I<em>s there snow in the background?”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/0*G9Pok8mnm_lCHxda.png" /><figcaption>A saliency map of where in the image the model is looking when being asked to answer if the animal is a wolf or a husky. Credit: “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, M.T. Ribeiro et al</figcaption></figure><p>The reason for this is that during training of the Neural Network model, the network was presented mostly images of wolfs with snow present in the background, while the huskies most frequently appeared in front of green vegetation. So the answers “<em>Yes/No</em>” turns out to be utterly meaningless since the question actually being answered is “<em>Is there snow in the background</em>” and not “<em>Is this a wolf?</em>”. Looking to the checklist of Interpretability, this clearly is a prime example of <strong>failing to ask the question correctly</strong> and thus leaving it up to the Neural Network to figure out which task it is supposed to perform.</p><blockquote>By definition, the purpose of a classifier model is to <strong>discriminate </strong>based on the features, or properties, of the data it is presented with. Here the snow offered a path of least resistance, as it was the most dominating feature available for the model to learn to discriminate the object classes by.</blockquote><p>Most progress with tools for Interpretability has been made with Neural Networks in Computer Vision, because the data is visual and hence these visual techniques are fairly low hanging fruit in terms of gaining intuition about a Network’s inner workings. As Machine Learning is becoming more and more prevalent in Decision Making systems based on non-visual data, like unstructured text or time-series data, or a combination of both, the need for an expanded tool box becomes evident.</p><h3>Q: Can the data answer the question?</h3><p>Predictably, there is an intense interest in leveraging Machine Learning to try to win the stock market by being able to forecast whether the price of stock will go up or down in the short term. This type of <a href="https://en.wikipedia.org/wiki/Regression_analysis">Regression analysis</a> on time-series data is a task that Machine Learning is well suited for, and it is fairly straightforward to train a Neural Network that learns to model historical stock price fluctuations and is able predict the next time step with some reasonable accuracy. However, the transition from a well-controlled testbed to the incredibly intricate and complex system of the uncountable and unknowable variables comprising the global trade markets, is about as smooth as a gravel path.</p><blockquote>A Machine Learning model cannot <strong>learn </strong>something it was never shown during training and it cannot know what it has never seen before.</blockquote><p>The model will fail when it encounters a never-before-seen occurrence in the data stream, in the sense that, while the model will always output <strong><em>an</em></strong> answer, the answer itself is nonsensical because<em> the question is ill-posed and lacks context.</em></p><p>Specifically, the model fails because in order for a Neural Network to be sufficiently robust to handle unexpected fluctuations in the stock exchange market, it has to learn basic human behavior and psychology. The model needs to be able to account for how people <a href="https://en.wikipedia.org/wiki/Groupthink">groupthink</a> and react to geopolitical events, natural disasters and other world encompassing calamities that perturbs the established and stable behavioral patterns in stock trades.</p><p>Ideally the model should also be able to read and comprehend press releases regarding staff changes in companies or to detect possible mentions in the media related to corruption scandals or future regulations that will affect earning potential. The problem of semantic analysis in Natural Language Processing is incredibly difficult and still unsolved, and until then, Machine Learning models cannot reliably distinguish between “good news” and “bad news” (or even “fake” news).</p><p>It is also expected that when word comes out that some actors are buying and selling on advice by an algorithm, there will be a positive feedback loop where herds of people will want to get in on the same perceived fool-proof trade, and situations may arise where there exists no incentive to sell a stock because after all, the AI-stock oracle said to buy it, not to sell.</p><p>All of these factors provides <em>the context</em> of a stocks valuation from moment to moment. The Black Monday crash of ’87 demonstrates how a crash can happen in a short span of time, with multiple players reacting to incomplete information from a variety of sources, such as policymakers, governmental agencies and the good ole rumour mill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*y4FwQMR2FmWjPpd5.png" /><figcaption>This timeline of the<a href="https://en.wikipedia.org/wiki/Black_Monday_(1987)"> Black Monday crash of October 19, 1987</a>, relating the sequence of events and changes in the S&amp;P 500 index, illustrates that simply numbers alone are not enough to understand fluctuations in the stock market.</figcaption></figure><p>So, let’s check in with the second criteria in the checklist, “Can the data answer the question?”. It is fairly straightforward to construct a dataset with the limited quantitative data regarding a specific stock, like, prices, performance, budgets, but this data will <em>not</em> give a full picture of the complex, hidden variables that affect how and why people decide to bet on a stock or not. And if <strong>the answer is not contained in your data</strong>, your model will not have the prerequisites for answering your question.</p><blockquote>There is currently no generic tool in the Interpretability toolbox to check the veracity and suitability of data, except our own critical thinking and healthy dose of skepticism.</blockquote><h3>Q: Can the answer be explained?</h3><p>Other, perhaps more noble, tasks for a Machine Learning system can be found within the healthcare technology industry, with use cases such as detecting anomalies in ECG data, abnormal blood values and other measurable quantities in the human body; to classifying objects in medical imaging such as tumors and fractures and so on.</p><p>These are all excellent Machine Learning problems. But when one attempts to use Machine Learning for automatic diagnosis and treatment suggestions based on a fairly limited set of data points pulled from a patients medical chart, things start getting hairy fast. The hairiness is not based on there not existing any underlying causation and correlation between groups of patients, diseases, symptoms, treatment regimens and so on, but rather on<strong> whether or not the data is sufficient</strong>.</p><p>As with the stock valuation example, we need to ask if the data contains all possible causes leading to the observed symptoms. Or are there important factors, unobserved variables, that are not quantified within the dataset? Or as with the “Wolf v Husky” example, is there a risk that the model has actually been trained to answer a different question than the one we <em>believe </em>we are asking it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/0*aVSlFMw6OE9N-i_W.png" /><figcaption>Currently, Machine Learning for use in medical diagnosing lacks transparency and verifiable results. ALEX HOGAN, I<a href="https://www.statnews.com/2017/09/05/watson-ibm-cancer/">KE SWETLITZ/STAT</a></figcaption></figure><p>Imagine that a specific disease is being treated at one particular hospital, with its in-house group of medical experts, serving a probably fairly homogeneous demography in one specific area of town. How can one guarantee that the model doesn’t actually classify the socioeconomic status of the patients and not their medical condition? That this model instead learns that heart attacks are only suffered by wealthy, white, middle-aged men in management positions while training for their next triathlon? Because that is the typical group of patients served by this particular treatment facility?</p><p>What happens if a company tries to sell a Treatment Recommender model like this to a clinic in a sub-Saharan country, where the majority of the patients are women with health issues related to being both female and poor, something that the model has never seen before? To check in with our list again: Can the physician trust the diagnosis and the treatment recommended by the Recommender model in this case?</p><p>The short answer is no, because she cannot open up the Black Box and <em>explain how</em> the model arrived at these recommendations. Choosing from a list of suggested treatment regimens sorted in order of how much “confidence” the models has, ranging from 0 to 100%, becomes an inane exercise without knowledge of the contribution and importance placed on the various input features, like symptoms, vital signs, gender, age, epicrisis, medical history and so on.</p><p>This case might sound like an hyperbole, but both the concerns and the dangers are real, because <a href="https://www.statnews.com/2017/09/05/watson-ibm-cancer/">Machine Learning for use in medical diagnosis</a> today lacks both transparency and proven safety and efficacy.</p><h3>First, do no harm</h3><p>A significant part of working as a Data Scientist comes down to expectation management and giving tempered advice on whether a Machine Learning system is appropriate for a given task, and also on proper and ethical use of data. Yes, Machine Learning does work! And really, really well, but with certain limitations. These limitations can be difficult to pinpoint precisely, as hopefully illustrated by the examples given in this article. But, being aware and able to identify limitations is key to doing Good Data Science, or at least, to avoiding doing harm.</p><p>In the end, we have to rely on critical thinking and following<a href="https://en.wikipedia.org/wiki/Precautionary_principle#Formulations"> the precautionary principle</a>: Avoid using technology that is not fully understood in Decision Making systems where the stakes are critical or high. While it is possible to train a model to achieve high accuracy for a specific and narrow task, knowing the why and the how a model arrives at a result is currently not really knowable due to the lack of standardized tools and approaches <strong>for interpreting and explaining.</strong> This is especially true for non-visual tasks.</p><p>What we can be certain of, however, is the maxim that if we put garbage in, we are going to get garbage out. And if the data is poor, either due to bad quality or low quantity, or maybe it can’t even corroborate the hypothesis in the first place, then we should have very low trust in the results and predictions made by the model that was trained on this data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27d1d1939b35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-accounts/good-data-science-interpreting-the-black-box-27d1d1939b35">Good Data Science — Interpreting the Black Box</a> was originally published in <a href="https://medium.com/tech-accounts">Tech Accounts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Democratized AI — The Black Box Problem]]></title>
            <link>https://medium.com/tech-accounts/democratized-ai-the-black-box-problem-d70145e4a606?source=rss-6f1e1b2254e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d70145e4a606</guid>
            <category><![CDATA[democratizing-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[accountability]]></category>
            <dc:creator><![CDATA[Hanne-Torill Mevik]]></dc:creator>
            <pubDate>Mon, 04 Mar 2019 10:01:01 GMT</pubDate>
            <atom:updated>2019-03-15T11:31:57.281Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uh5eLQY7LhXtGN_u-jb_3w.png" /><figcaption>Illustration by <a href="https://hanne-morstad.myportfolio.com/">Hanne Morstad</a></figcaption></figure><h3>Democratised AI — The Black Box Problem</h3><pre>This is the first post in the series &quot;How To Good Data Science&quot;, which centers around Interpretable Machine Learning and related topics.</pre><p>Can you imagine modern life without electricity? I bet the idea seems ludicrous, and clearly, if you are reading this, you are <a href="https://data.worldbank.org/indicator/EG.ELC.ACCS.ZS">among the 87% of the world</a> population that enjoys access to electrical power. Andrew Ng, a household name in the Machine Learning community, famously stated that “<a href="https://twitter.com/andrewyng/status/735874952008589312?lang=en">AI is the new electricity</a>”, which is a strikingly accurate analogy when one starts unpacking its implications.</p><p>Because, while electricity mostly produces perceptible, understandable and tangible effects, the results of using AI, or to be precise, Machine Learning technology, is innocuous, inscrutable and hidden for most of us. Nevertheless, Machine Learning pervades most aspects of modern society and industries, fueled by incentives such as efficiency, optimization, cost savings, increased insight and productivity.</p><p><a href="https://medium.com/bakken-b%C3%A6ck/how-biased-ai-can-teach-us-to-be-better-humans-3be60ebbe558">There are several pitfalls to be aware of</a>. Arguably, the worst outcome is <em>not</em> having wasted time and money on implementing a Machine Learning system that didn’t perform as well as hoped. It would be far worse if this system participates in perpetuating societal bias that hurts people and harms our society, either surreptitiously through discrimination; By limiting equal access to opportunities; By declining mortgages with no possibility of appeal; Or by hiding job listings from candidates that don’t fit the traditional profile.</p><p>Alternatively, the system could cause overt injury, through for instance autonomous vehicles that make fatal decisions based on poorly understood image recognition, or by recommending medical diagnoses and treatments to patients that are derived from flawed reasoning.</p><h4>There’s a little Black Box for everyone</h4><p>Progress in Data Science and Machine Learning and runaway freight trains have a few things in common, like speed and a chance of accident waiting to happen. Academic researchers are frantically spitting out papers featuring new algorithms and incrementally improved results. Meanwhile, even swifter progress is occurring on the hardware and software side of the same AI hype-coin. Few others, if any, research fields have so rapidly transitioned from being a small niche of mostly academic interest, to becoming so readily available and desirable to learn for non-expert users.</p><p>This trend is fuelled by a fortuitous congruence of circumstances. The essential programming languages for Machine Learning are intuitive and high-level, and can easily be learned in a few weeks time by someone already familiar with computer programming. And, with the availability of powerful open-source libraries and frameworks, coupled with high quality data sets in the Public Domain and cheap cloud computing, Machine Learning has turned into a fairly low entry endeavour.</p><p>But these technical aspects aside, an inherently wonderful benefit of Machine Learning being a young research field is that the general rule from the onset has been to publish scientific papers and results openly — as pre-prints on <a href="https://arxiv.org/">arXiv.org</a> and in <a href="https://en.wikipedia.org/wiki/Open_access">Open Access</a> journals and proceedings such as <a href="http://www.jmlr.org/">JMLR</a> and <a href="http://proceedings.mlr.press/">PMLR</a>. Tutorials and lectures are freely shared on YouTube and as free online university courses. This has led to a democratisation of Machine Learning knowledge, where skills and know-hows really only depend on access to electricity, internet and a laptop. In other words: <em>Anyone </em>can do Machine Learning today.</p><p>However, the mere fact that anyone and their grandma<em> are able </em>to get a Machine Learning system up and running, does not necessarily imply that there is <em>any understanding, </em>or even intuition, of what is going on under the hood, neither during development nor after the system is put in production. Most such systems are magic black boxes, where one inserts data into one end, stirs around for some indeterminate amount of time, and then a number plops out on the other side.</p><p>So, how can we ward ourselves from potential harmful outcomes if we don’t know what is going on? Is it possible to peek inside this Magic Black Box that is Machine Learning, and understand how and why a Machine Learning system arrived at a particular result?</p><h4>How to Good Science</h4><p>Let’s take a step back and contemplate the word “science” that makes up one half of “Data Science”. It is hard to properly define what constitutes “Good” science, but, if we look to the STEM research fields, there are some established guidelines and principles to follow in order to avoid doing outright bad science. During the designing of an experiment to test an hypothesis, it is of equal importance to put considerable thought into how to <strong>ensure an optimal execution</strong><em> </em>of the experiment, by following these steps:</p><blockquote>Isolate the experimental setup to avoid contamination from external influences.</blockquote><blockquote>Choose proper metrics.</blockquote><blockquote>Accurately measure and store the sampled data.</blockquote><blockquote>Analyse and interpret the data.</blockquote><blockquote>Verify the results.</blockquote><p>Especially important is The Zeroth Law: Remember to leave predetermined viewpoints at the door, and emotionally prepare for having the hypothesis completely disqualified…</p><p>These guidelines are applicable to Machine Learning as well, if one thinks of the training of the model as being the experiment, and the resulting output by the model as being the data sampled and collected during the execution of the experiment. Choosing suitable metrics, storing the output, and even verifying the results, are easily transferable concepts.</p><p>The challenge lies with <em>controlling for outside influences </em>and <em>interpreting the output.</em> Because the training data that is put into the Machine Learning system contains both the <em>desired signal </em>that tests the hypothesis, but also <em>external noise</em> that contaminate and confounds the result. And, when interpreting the result, we need to be able to decouple the signal from the noise.</p><h4>Good <em>Data Science</em> requires Interpretability</h4><p>Academic and corporate interests have been more focused on advancing the theories, algorithms, and software tools in order to perform increasingly more elaborate experiments, rather than on experimental verification, as in the more established physical sciences. Fortunately, the need for accountability is catching up, germinating into the sub-field of <strong>Interpretable Machine Learning</strong>.</p><p>Interpretability in Machine Learning is not a well defined concept, because it is<a href="https://drive.google.com/file/d/1jYcca3BsNE7IEgDc5gZV-gfEwxuaQLsY/view?usp=sharing"> context dependent</a> and will mean different things for different problems. But, arguably, one can take it to mean opening up the magic black box, rendering it <em>transparent</em>, and being able to<strong> interpret, explain </strong>and <strong>trust </strong>what is going on inside it. Speaking in general terms, to assess if one has achieved Interpretability, I propose that we should be able to answer the following questions with some degree of confidence (that is, given our hypothesis, our questions posed to the data):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOcdhvsS3yaNhmZVyfkRnw.png" /></figure><p><a href="https://medium.com/tech-accounts/good-data-science-interpreting-the-black-box-27d1d1939b35">In part two in this series</a> I will embellish on this checklist and supply examples to argue the relevance of these questions and outline potential outcomes of failing to answer them.</p><h4><strong>Understanding data is easier than understanding science</strong></h4><p>It might be tempting to buy into the pitch that <em>“Your own engineers and developer teams understand your data the best, </em><a href="https://singularityhub.com/2018/02/19/the-democratization-of-ai-is-putting-powerful-tools-in-the-hands-of-non-experts/#sm.00013bxzzru1ef07uxb2389v8ni1i"><em>so just re-skill them</em></a><em>”</em>. Sure, <em>ideally </em>every company that has some data-driven operational aspect, or provides data-driven services, should have an in-house team of data scientist and Machine Learning experts. Also, due to the scarcity and high demand of this competency, it makes sense to retrain in-house developers.</p><p>A common misconception is that most data scientists are <em>not</em> programmers,<em> </em>but often has a Master’s Degree or PhD in STEM fields, such as Mathematics, Statistics or Physics, with years of experience in doing “Good Science”, and have acquired a generally <a href="https://en.wikipedia.org/wiki/Skeptical_movement#Scientific_skepticism">sceptical attitude</a><em> </em>towards both data quality and model veracity. However, if one does decide on going down the retraining road, it is highly recommended that the team adheres to the Checklist of Interpretability and becomes equipped with the following:</p><blockquote>The knowledge required to identify issues and shortcomings of the data.</blockquote><blockquote>The skills to properly pose the questions that one wants the data to answer.</blockquote><blockquote>The abilities to interpret and explain the answers the system outputs.</blockquote><p>And lastly, for everyone involved, including the product owner, to gain a deep appreciation for <a href="https://en.wikipedia.org/wiki/Precautionary_principle#Formulations">The Precautionary Principle</a>:</p><blockquote>To avoid using Machine Learning technology that is not fully understood in <strong>decision making systems where the stakes are high or critical.</strong></blockquote><p>Getting Machine Learning education and training right will play a significant role in the improvement of our collective future. Achieving the common goals of halting climate change, fighting poverty and raising the living standards in the developing countries, requires us to make technology that is resource and energy efficient. It must also be interpretable, explainable and proven worthy of our trust, so it does not cause unintended harm along the way.</p><p>After all, who would accept reduced access to electricity in order to lessen their ecological impact? Would you?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d70145e4a606" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tech-accounts/democratized-ai-the-black-box-problem-d70145e4a606">Democratized AI — The Black Box Problem</a> was originally published in <a href="https://medium.com/tech-accounts">Tech Accounts</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How biased AI can teach us to be better humans]]></title>
            <link>https://medium.com/bakken-b%C3%A6ck/how-biased-ai-can-teach-us-to-be-better-humans-3be60ebbe558?source=rss-6f1e1b2254e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3be60ebbe558</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hanne-Torill Mevik]]></dc:creator>
            <pubDate>Tue, 27 Feb 2018 15:38:39 GMT</pubDate>
            <atom:updated>2018-05-25T09:42:42.568Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X3TNZbdu06LWNCpHixIFxQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/0E_vhMVqL9g?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Andy Kelly</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Let me tell you, now is an exciting time to be a data scientist with everyone and their grandmothers jumping on the AI bandwagon, all wanting <em>an AI to fix all the things. </em>I am immensely enjoying the <a href="https://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century">new sexy status</a> of the historically nerdy field of Machine Learning. In case you’re not familiar with that term, you can thank PR and marketing people for somehow managing to re-brand the messy intersection where Statistics, Math and Computer Science meet and merge into the much hyped term “Artificial Intelligence”.</p><p>Now, before you head on out there, eager to <a href="https://medium.com/bakken-b%C3%A6ck/a-year-with-orbit-operator-13442444bbdf">build your own chatbot</a> and release it into the wild, there are a few lessons to take to heart. Aside from the feasibility and the usefulness of the tasks we can automate today with machine learning (that’s a whole post in itself!) there’s another, more sinister, side to the story. I will share with you a cautionary tale of <em>machine bias</em><strong> </strong>and how it <em>already is </em>affecting you today and <em>why you should care</em> that mitigating measures are taken to either prevent it when possible or lessen its effect. If you make it to the end of this post, you will gain an understanding of why <em>de-biasing</em> is such a difficult task.</p><p>Firstly, we should agree on <em>what </em>bias exactly is. Bias can mean many things, but in the scope of this post we turn to <a href="https://en.wikipedia.org/wiki/Bias">Wikipedia</a> for two specific definitions. Bias exhibited by humans is often called <a href="https://en.wikipedia.org/wiki/Bias#Cognitive_biases">Cognitive bias</a>,<strong><em> </em></strong>meaning</p><blockquote><em>a repeating or basic misstep in thinking, assessing, recollecting, or other cognitive processes.</em></blockquote><p>When the outcome of such an illogical reasoning is negative, this type of bias is often referred to as <em>prejudice</em>.</p><p>The honest truth is that what we currently call “AI” isn’t capable of actual <em>cognition of any kind, </em>and the missteps in an AI’s programming logic arise from buggy code written by an overworked programmer. Rather, the <a href="https://en.wikipedia.org/wiki/Sampling_bias">bias we observe</a> in machines is caused by</p><blockquote><em>systematically favouring some outcomes over others, or by using data consisting of an unfair sampling in the population</em>.</blockquote><h3><strong><em>A hypothesis</em></strong></h3><p>I propose the following hypothesis, that<strong><em><br></em></strong><em>1. Unfair data is the quantitative expression of cognitive and societal bias, and <br>2. Machine bias is a product of unfair data.</em></p><p>It follows that unmitigated or unchecked machine bias can perpetuate societal prejudice, but also that <strong>machine bias has the potential to be a tool to uncover biases and prejudices we are unaware of</strong>.</p><p>In this post I will give examples that support my hypothesis and argue why you should care. Most of these examples originate in the U.S., where industry giants like Google, Facebook, Amazon, Microsoft, Apple and Baidu are drawing on near unlimited resources and pushing the envelope of research and development and the adoption of Machine Learning into products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*LwYtpmqjs3P0dEWJ.gif" /><figcaption>Releasing your AI thinking it is the shit</figcaption></figure><p>In Norway, and the Nordic countries in general, we are lagging a fair bit behind when it comes to incorporating machine learning into educational programs and commercial products. Now is therefore an opportune time to learn from the mistakes of others and avoid the common pitfalls.</p><h3>The pitfallacy — What’s in a “data”?</h3><p>Let’s start with an association game around the word “data”. Some of you will think of numbers and statistics, something that is well defined, well known, static and immutable, impartial, fair or unfair in its inability to adjust to mitigating circumstances. Others among you will think of Star Trek’s Data, as many of the words I listed are also counted among that character’s personality traits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*Emj76yW8grFe3b8S.gif" /></figure><p>Our assumption is that if we feed our <em>believed-to-be impartial data</em> into a machine learning algorithm — in order to train a <em>model</em> to solve some task for us — then this resulting model will be equally as impartial and rational and fair as the input data.</p><p><strong>The fallacy is that data <em>is not at all </em>impartial! </strong>It is equally as biased and prejudiced as us humans who generated it. You can be sure that all the -ism’s are represented: Sexism. Ageism. Racism. Ableism. Heterosexism. Nativism. Lookism. (Yup, I had to look some of them up.) When the data is skewed towards some feature, the resulting machine learned model will be equally skewed.</p><p><a href="https://arxiv.org/pdf/1608.07187.pdf">Research </a>shows that when applying standard and widely used Natural Language Processing tools to our written languages, machine learning algorithms will record and display the same biases that humans demonstrate in psychological studies. This happens because <a href="https://en.wikipedia.org/wiki/Semantics">semantics</a>, the meaning of words, necessarily reflect regularities latent in our cultures, some of which we know to be prejudiced.</p><h3>Bias embedded in the data</h3><p>Take <a href="https://www.theguardian.com/commentisfree/2017/apr/20/robots-racist-sexist-people-machines-ai-language">gender stereotypes found in written texts </a>as an example. Our language is the lens through which we learn to understand the world, its populations and their stories. We learn that the word “man” in both Norwegian and English, can refer to a male individual or “all of humanity”, but the word “woman” never does. The narrative in my math and physics books from high school and university was always male, using a “he” to either describe a man, someone in general or the reader, and never a “she”. (In recent years an effort is being made to use a more equally gendered language.) It is therefore no coincidence that the machine learned models — which are trained on texts and articles scraped from Twitter, Wikipedia, books and news articles — will learn these same gendered semantic patterns that exist in our historical and current ways of expression.</p><p>The <a href="https://code.google.com/archive/p/word2vec/">Word2Vec algorithms</a> <a href="https://arxiv.org/abs/1301.3781">published in 2013</a> quickly achieved popularity for combining good performance with a fairly understandable representation of words as numbers. The models learn the linguistic context of words and their semantic relationship in an <a href="https://en.wikipedia.org/wiki/Unsupervised_learning"><em>unsupervised fashion</em></a>, meaning that they receive no instructions, nor do they have any prior knowledge about the language.</p><p>One fun property of these language models is that we can query them for linguistic analogies and similarities. We can ask questions like: “Paris is to France, what Oslo is to &lt;blank&gt;” and receive the answer Norway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/0*MKdvH0AIwdIwIQbW.png" /><figcaption>Illustration of how Word2Vec learns semantic relationships between different types of words. Source: <a href="https://www.tensorflow.org/tutorials/word2vec">https://www.tensorflow.org/tutorials/word2vec</a></figcaption></figure><p>We also find that <a href="https://arxiv.org/pdf/1607.06520.pdf">“she” is to “nurse” what “he” is to “doctor”</a>. And that “female” and “woman” are more closely associated with arts and humanities occupations and with the home, while “male” and “man” are closer related to math and engineering professions. Or that pleasant words like<a href="http://students.washington.edu/adelak/2017/01/?p=183"> “gift” or “happy”</a> are placed in context with white American names, while <a href="https://en.wikipedia.org/wiki/African-American_names">black American names</a> are more associated with unpleasant words. This shouldn’t come as a surprise since there is already plenty of public debate about bias concerning foreign or ethnic sounding names, as exemplified by two, all too common and unfortunate practices, in which job applicants won’t get called into interviews or house seekers have a difficult time renting an apartment, both due to having Arabic sounding names. These are a few examples of how the <em>bias embedded in the data</em> leads to machine bias: that biased semantic patterns in our language are expressed in a learned mathematical model.</p><h3>Bias because of missing data</h3><p>On the opposite end of the spectrum, machine bias in Computer Vision and object detection in images commonly arise due to a <em>lack of data</em>, as the dataset of images used to train the model does not include all possible variations of the object you want it to detect. Evidence of this is found in some quite cringe-inducing examples. For instance, there was the time Google Photos auto-tagged <a href="https://www.theguardian.com/technology/2015/jul/01/google-sorry-racist-auto-tag-photo-app">black people as gorillas</a>, with <a href="https://www.theguardian.com/technology/2015/may/20/flickr-complaints-offensive-auto-tagging-photos">Flickr following suit</a>. And then Nikon’s Face Recognition software couldn’t<a href="https://thesocietypages.org/socimages/2009/05/29/nikon-camera-says-asians-are-always-blinking/"> deal with Asian features</a>. Machine bias due to lack of varied data is not race-exclusive, but can also <a href="https://singularityhub.com/2017/01/31/the-struggle-to-make-ai-less-biased-than-its-creators/">include ageism</a>. How about the app that <a href="http://beauty.ai/">tries to classify beauty</a>, that associates only images of young, white people with beautiful faces? Or the one that <a href="http://aging.ai/">attempts to predict age</a> based on blood tests, failing miserably when used on an older population?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*L7aQyKNzp5bv3exB.png" /></figure><p>To be more specific, the racist bias exhibited by Google Photos, occurred because the algorithm was shown only faces with similar, lighter skin colour during the training of the object detection model. In addition to learning to recognise edges and colour changes that indicate eyes, mouths, nostrils and teeth, the model also learned from that unvaried data set, that a light skin color is an equally important feature for accurately detecting human faces. If, instead, they had added training data containing all possible variations of skin tones across the entire human populace, the model would have learned to put <em>less emphasis on the skin</em>, and more emphasis on the presence of two eyes, one mouth, nose, chin and forehead.</p><p><strong><em>Actually, this is the main takeaway lesson from this post:<br></em></strong> If your intuition is telling you that there is a feature or property in your data that shouldn’t be as important as others: then either remove it entirely, if you can, or make sure to include it in its entire spectrum of possible values to <em>lessen its effect </em>in the learned model.</p><h3>Bias from reading between the lines</h3><p>What if your data is neither unstructured texts nor images, but rather <em>structured textual data</em> in the form of personal information, be it name, gender, age, address, income, educational background, profession, sexual preference, political affiliation? Please pause for a moment here and appreciate just how much personal information you have disclosed to for-profit companies, like your bank, phone service provider, insurance company, Facebook, LinkedIn, grocery store loyalty programs. The list goes on. All of these companies have incentives to build recommendation systems based on your data (or to sell them to someone who does) to create or offer services specifically tailored to suit you. The stakes of certain systems are higher than others; arguably your life will be more impacted by a declined house loan, than say, a poor book suggestion on Amazon Kindle.</p><p>One infamous and alarming example of a machine learned prediction system used in high-stakes decision making comes from the American criminal courts. A commercial software called COMPAS was employed by judges to predict how likely a person charged with a crime was to re-offend at a later time. The calculated risk score was used to determine the sentencing, which could mean offering the defendant a plea deal or a prison sentence of several years. An extensive analysis was performed by <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">ProPublica</a>, who found that the risk assessment was heavily biased against the African-American community.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/0*X3WDt82TXFZUUHbm.png" /><figcaption>Source: <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">ProPublica</a>. Examples of risk scoring of a white male and a black, young adult female.</figcaption></figure><p>The article explains that the practice of using race, nationality and skin colour making recidivism predictions ended in the 1970’s, when it became politically unacceptable. So where did the machine bias come from?</p><p>COMPAS calculated a score derived from <a href="https://www.documentcloud.org/documents/2702103-Sample-Risk-Assessment-COMPAS-CORE.html">137 questions</a> that were either answered by defendants or pulled from criminal records. Race was not one of the questions. However, the survey asked defendants questions such as: “Was one of your parents ever sent to jail or prison?”, “How many of your friends/acquaintances are taking drugs illegally?” and “How often did you get in fights while at school?”. The questionnaire further asked people to agree or disagree with statements like “A hungry person has a right to steal” and “If people make me angry or lose my temper, I can be dangerous”.</p><p>A validation study of the risk of recidivism score calculated by COMPAS showed an accuracy of 68 percent. But when questions and data items that could be correlated with race — such as poverty, joblessness and social marginalisation — where omitted from the data, the accuracy went down, to hardly better than a coin toss. It is easy to draw up a picture of one “rotten apple” in the fruit basket that is the white, working-middle class, who would answer no to the questions mentioned above, yet still be likely to commit new crimes later on. In contrast, considering that<a href="http://www.apa.org/pi/ses/resources/publications/minorities.aspx"> 39 percent of black American </a>children and adolescents and 33 percent of Latino children and adolescents are living in poverty — compounded with unemployment rates twice that of white Americans — a high risk score would automatically be attributed a black or Latino individual simply by virtue of being born.</p><p>In the case of COMPAS, the failings are on multiple levels and not simply caused by using a machine learned model based on biased data. There is the misuse and appropriation of a tool for sentencing, that was originally intended to assist in selecting candidates for suitable rehabilitation programs. Secondly, there is the non-trivial challenge of asking questions that are neutral to socioeconomic status. And lastly, there is the problem of perpetuating social bias through unnecessarily harsh punishment of offenders, causing them to have problems with reintegration into society with a criminal record they didn’t deserve in the first place.</p><h3>Is your name Acceptable?</h3><p>Fortunately, the American criminal court is on the more extreme end of the spectrum, and most of us will hopefully not be the object of its attention. However, j<a href="https://www.theguardian.com/news/datablog/2013/aug/14/problem-with-algorithms-magnifying-misbehaviour#two">ob matching</a> is a much more innocuous high-stakes recommendation system. Recruiting agencies and head hunters are now automating the initial phase of matching profiles with job listings. Almost all of them use<a href="https://www.forbes.com/sites/georgeanders/2013/04/10/who-should-you-hire-linkedin-says-try-our-algorithm/"> LinkedIn’s Recruiter tool </a>during some part of the process, a tool which employs machine learning to recommend profiles to a job listing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*yblVYL8C5Qvhf3UX.png" /><figcaption>Source: <a href="https://www.forbes.com/sites/georgeanders/2013/04/10/who-should-you-hire-linkedin-says-try-our-algorithm/">https://www.forbes.com/sites/georgeanders/2013/04/10/who-should-you-hire-linkedin-says-try-our-algorithm/</a></figcaption></figure><p>LinkedIn actually does <em>not</em> ask for information regarding gender and ethnicity. But, if you’ve managed to read this far, you now understand that textual data carries semantic meaning and relationships that can be learned by mathematical algorithms. Our names are gendered, our educational backgrounds and resident cities are correlated with ethnicity, school names are related to socioeconomic status, and previous work experience and job titles will be linked with the future possible matching with new job listings.</p><p>Other market places for job listings employ a click-based payment model, in which the amount of clicks on a listing directly affects earning potential. Their models exhibit a <em>conflicting goals bias</em> caused by the monetary incentive to <a href="https://techcrunch.com/2016/12/10/5-unexpected-sources-of-bias-in-artificial-intelligence/">show jobs that fit people’s self-view,</a> thereby perpetuating societal bias. Women, for instance, are shown to be more likely to click on an ad for a position related to “nursing” rather than one that advertises for a “medical technician”.</p><p>The banking industry serves as another example of non-transparent, high-stakes decision making regarding housing loans. Several banks that operate in the Nordic market have already started to supplement, or even replace, the traditional credit score with machine learned models in order to decide whether to grant applications for housing loans. They use their internal client data to train a model that predicts how likely future clients are to keep up with mortgage payments. Surprisingly, some banks are actually experiencing an increase in the number of loans granted, so hopefully this practice turns out to be a win-win for all parties involved and not the beginning of the next <a href="https://en.wikipedia.org/wiki/Credit_crunch">credit crunch</a>.</p><h3>Bias-skepticism is for Everyone</h3><p>There are several reasons for why you should care about Machine bias. You could be motivated by…</p><ul><li><em>Idealism</em>: you want to make the world a better and more inclusive place;</li><li><em>Professional interest</em>: your company or product might not survive negative publicity;</li><li><em>Altruism</em>: you are concerned about fair availability of opportunities and resource allocation;</li><li><em>Truth seeker</em>: you want the <em>truest, bestest</em> data that can be;</li><li><em>Personal gain</em>: you want access to opportunities, education, employment, health care</li></ul><p>… <strong>because eventually you will get a score, too.</strong></p><p>Whichever category of motivation you fall into, in all likelihood you will be affected by machine bias in one way or another. It can be blatantly obvious, like the Facebook filter bubble, or it could be more insidious and hidden within job listings that are never offered to you, because your name fits a pattern of minorities.</p><p><strong>Or we could flip the problem of machine bias on its head:</strong></p><p>Perhaps the medical treatment that you need is not suggested to you, because treatment based on ethnic traits is considered to be a topic that is too difficult, too taboo and too politically dangerous for allowing medical research into. Perhaps machine learning techniques can be used to uncover systematic differences in drug and treatment efficacy within different ethnic groups in a more palatable way.</p><h3>We can do better</h3><p>As beings with natural intelligence we are equipped to be self-aware and consciously counteract our own learned cognitive biases, but the algorithms available to us presently are not incorporating bias-adjusting measures. Fortunately, gauging by the unprecedented amount of papers, keynotes and general chatter about bias and fairness at the Conference on Neural Information Processing Systems of 2017, much attention is now being directed towards developing de-biasing techniques and methods.</p><p>There are already some simple steps we can take to limit the effect of bias on our data. For instance, we can synthetically augment the data used to train our models. This is an established, non-controversial practice for generating sufficiently large and varied data sets that are suitable for learning. Essentially this involves taking samples from the real data, then performing small changes to the samples before adding these new, altered samples to the data set. For image data, the alteration can be adding noise, or shifting, scaling, rotating or flipping the image. Changes in skin tones and contrast settings could be a way to make the algorithm less sensitive to colour as a feature.</p><p>For text data, an analogous approach exists, where synthetic sentences can be added to the data by randomly replacing words with synonyms taken from a dictionary. One possible augmentation could be to add sentences in which male gender nouns are replaced with female, or Nordic-European sounding names are switched with names of different ethnic origins. Recently,<a href="https://blog.conceptnet.io/2017/04/24/conceptnet-numberbatch-17-04-better-less-stereotyped-word-vectors/"> mathematical techniques have been proposed</a> to reduce the effect of gendered language by either adjusting the numbers before or after the prediction is made.</p><p>Finally, try to be brave and look your biased AI straight in its binary face. All of our collectively flawed humanity is looking back at you, and it’s not pretty. Review your company hiring practices (Check out the <a href="http://wimlworkshop.org/directory-of-women-in-machine-learning/">directory of women working in machine learning</a> and <a href="https://blackinai.github.io/">Black in AI</a>.)<a href="https://www.technologyreview.com/s/610192/were-in-a-diversity-crisis-black-in-ais-founder-on-whats-poisoning-the-algorithms-in-our/"> When the workforce is diverse,</a> your colleagues or employees will bring perspectives and experiences that will help you check your bias before implementing and releasing that AI into the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*GtAZUuZYKwaqZ5d4.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3be60ebbe558" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bakken-b%C3%A6ck/how-biased-ai-can-teach-us-to-be-better-humans-3be60ebbe558">How biased AI can teach us to be better humans</a> was originally published in <a href="https://medium.com/bakken-b%C3%A6ck">Bakken &amp; Bæck</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>