<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sidarth Jayadev on Medium]]></title>
        <description><![CDATA[Stories by Sidarth Jayadev on Medium]]></description>
        <link>https://medium.com/@sidjayadev97?source=rss-cb567727317e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Sidarth Jayadev on Medium</title>
            <link>https://medium.com/@sidjayadev97?source=rss-cb567727317e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:46:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sidjayadev97/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Are Large Language Models Too Large?]]></title>
            <link>https://pub.aimind.so/are-large-language-models-too-large-909367e5dbee?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/909367e5dbee</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[probabilistic-programming]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Tue, 22 Aug 2023 18:20:41 GMT</pubDate>
            <atom:updated>2024-02-29T23:00:04.044Z</atom:updated>
            <content:encoded><![CDATA[<h3>Are Large Language Models really human-like?</h3><h4>Exploring NLP hierarchically instead of sequentially</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d-0m_dYtV72w8aOt" /><figcaption>Photo by <a href="https://unsplash.com/@ilgmyzin?utm_source=medium&amp;utm_medium=referral">ilgmyzin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Recently, I’ve been seeing the media hype and buzz surrounding ChatGPT. A very similar type of hype-bubble surrounded GPT-3 just a short while ago — and obviously it makes sense. These language behemoths could imitate history’s greatest poets, write formal essays, spit out seemingly intelligent responses to human questions, and even write programming code (albeit with some errors.) I actually made an OpenAI account a few months back and tried to ask ChatGPT about mental health, mathematics, and even about itself — and the responses are honestly amazing and very human-like. However, after a while it’s not too hard to see where the artificial side of language-generation seeps into the responses, but ChatGPT still created a mini-existential crisis in my stomach.</p><p>Now the answer to the title might seem “obvious”, since ChatGPT isn’t a computer — but this question really requires deeper analysis.</p><p>Speaking to the artificial side — as before with GPT-3 — ChatGPT is also starting to show its mistakes. It has huge social biases in the model &amp; it’s potentially environmental un-sustainable, as Medium author <a href="https://towardsdatascience.com/not-all-rainbows-and-sunshine-the-darker-side-of-chatgpt-75917472b9c">Dr. Mary Reagan points out in her amazing article</a>. And <a href="https://analyticsindiamag.com/freaky-chatgpt-fails-that-caught-our-eyes/">Chat-GPT doesn’t really understand language like us, fails at logical reasoning, can convince users but be totally wrong, and has many other similar flaws.</a> Another important issue that isn’t directly related to the model flaws is just the sheer amount of sentences and language data ChatGPT was trained on. Financially speaking, <a href="https://www.sciencefocus.com/future-technology/gpt-3">the model was trained on 570 GB of data</a>, and costs approximately <a href="https://www.businessinsider.com/how-much-chatgpt-costs-openai-to-run-estimate-report-2023-4">$700,000/day to operate on specialized servers</a>. Let’s not also forget the infamous fact that, <a href="https://time.com/6247678/openai-chatgpt-kenya-workers/">as reported by <em>Time</em>, OpenAI exploited Kenyan workers for pennies</a> to mine enough training data to minimize toxicity in ChatGPT‘s generative text.</p><p>And finally, a language/image-data analysis by machine-learning experts at MIT and Epoch suggest that <a href="https://arxiv.org/pdf/2211.04325.pdf">humans might run out of high-quality language data by 2026</a> — not that far away.</p><p>So then, what’s the connection between inefficiently-large training data, and what might large language models be overlooking? These large-language models (LLM) are usually based on something called a “Transformer” — a type of neural network that looks at words sequentially and generates words in order sequentially. Transformers are a successor to recurrent neural networks (RNN), and have a lot of advantages over RNNs with better mechanics like attention, and self-attention to retain a better memory of previous words.</p><p>But sequential analysis of natural language, might be just the issue. We are training LLMs on a half-terabyte corpus with billions of parameters, is because the words in a sentence are being analyzed <strong><em>sequentially. </em></strong>The model is basically building up large statistical distributions of thousands, millions of n-grams, word-vectors, contexts and more — but in a linear way. And to get a better grasp of natural-language AI, I argue we should <em>not</em> be creating models that produce sentences and words like this. Why? Because of this simple fact:</p><blockquote>Language has <strong>hierarchical structures</strong> that can be efficiently and directly exploited. It’s not clear if LLMs do this directly. They might infer it after millions of sentences, but could they do it with just a few hundred?</blockquote><p>There’s a subtle nuance to discuss here: while LLMs are generative models, their structure might not make them amenable to being pre-trained with low amounts of data. LLMs are great at teasing out statistical patterns in sentences, seeing which words follow which according to topic context. But this doesn’t seem like a human-like AI; humans don’t usually speka or write sentences by considering probabilities of successive words or punctuations. Furthermore, LLMs don’t seem able to tease out the <em>underlying reason</em> for these statistics. In that sense, it’s not truly a “generative model”; it can definitely generative the surface statistics, but it’s missing a core aspect of natural language sentences — hierarchy. And the easiest type of sentence hierarchy to explain, is called a “syntax tree”. It’s quite abundant in English (formal or not!), and exists in virtually every programming language.</p><h3>Syntax Trees</h3><p>What are syntax trees? In natural language (for example, English), sentences have a hierarchical-like structure to them that grants a successive level of abstraction — even if the sentence itself has many words in it. Here’s an example called “Phrase-structure grammar”, or constituency-grammar, first coined by the linguist Noam Chomsky:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*5sBzu9PC3pbKX5IPbsWzVQ.jpeg" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Parse_tree">https://en.wikipedia.org/wiki/Parse_tree</a></figcaption></figure><p>Let’s dissect this. <em>S</em> of course, is the sentence. <em>N</em> is a noun, but <em>VP</em> is a verb phrase, which will consist of a verb, along with other things. We can see the “first level of abstraction”: a sentence in English should consist of a noun phrase and a verb phrase. The 2nd level of abstraction is that verb phrases consist of a verb + noun phrase. And the 3rd level shows us that noun phrases are a definite article (D) + noun (N).</p><p>In current sequential models such as BERT, GPT-3 and ChatGPT, the model-creator must worry about 1) memory &amp; attention to words/characters many spaces away, and 2) the unbelievable training-data sizes. But by understanding the levels of abstraction such as noun/verb-phrases and clauses and their order within the sentence, it would be easier to generate sentences with <em>much</em> less data. Why?</p><p>Let’s look at the sentence diagram above. Using abstraction, we can see hierarchically that the sentence consists of a (N, V, D, N) ordered pair — a noun, verb, article, and noun. Say we have a bag-of-words set from a topic with 20 nouns, 20 verbs, and 5 articles. To find the number of grammatically correct sentences, we just calculate the product 20 x 20 x 5 x 19 = 38,000 sentences. We could generate 38,000 grammatical sentences from only <em>45 words</em>. Now that is data-efficient. Would these sentences make any meaningful sense <em>semantically</em>? If the bag-of-words are chosen from a topic mode, perhaps! If otherwise, they probably wouldn’t work semantically. Preserving semantic meaning is definitely important, but so is data efficiency.</p><p>There is a big elephant hiding in the room, however: for many languages the AI system isn’t given the syntax tree beforehand. So this leads us to the follow-up question:</p><h4>Can a machine infer syntax trees?</h4><p>Syntax trees are great, and generative models that spit out grammatically correct sentences from small data without sequential analysis, sound cool. But if we want the computer to truly “self-learn”, we need the computer to infer the syntax-tree all by itself, before using it generatively. How would it do that?</p><p>In computational linguistics this problem is called<em> grammar induction</em>. And as recently as August of 2022, cognitive and computer scientists have been trying to solve this problem. Back in August, MIT scientists Josh Tenenbaum, Kevin Ellis and others were <a href="https://www.nature.com/articles/s41467-022-32012-w">approaching grammar induction using a probabilistic framework called Bayesian Program Learning, or BPL.</a> Even 30–40 years ago, this problem was being studied, with various computational approaches, such as <a href="https://aclanthology.org/P95-1031.pdf">this early form of probabilistic grammar induction</a> by Stanford computer scientists.</p><p>Inferring syntax trees is only one part of the puzzle, and another issue in that is the consistency of the tree, and how far natural language deviates from it. MIT’s approach to language modeling was very different from the usual large-language models; they made use of a key core idea: the <em>child-as-linguist</em> analogy. BPL (Bayesian Program Learning) is just a fancy word for inductive-inference ; the basic idea is that BPL infers a generative procedure, represented as a <em>symbolic program </em>using probabilistic programming languages. In the context of natural-language processing, BPL infers a symbolic program (using Bayes’ Theorem, hence the name) that can generate grammar. And one of the essential motives for using this approach is that BPL methods, while new to AI, are much more data-efficient to train than LLMs.</p><h3>The Best of Both Worlds?</h3><p>However, LLMs already exist, so we can’t really turn back the clock. So are we doomed to never be able to create true, human-like, efficient natural language AI?</p><p>Not so fast.</p><p>It might be more pragmatic to develop effective natural-language systems by <em>combining</em> existing, pre-trained LLMs with commonsense probabilistic reasoning — instead of current NLP ventures: non-stop production of more and more complex (and larger) LLMs, incurring (potentially) large opportunity costs and diminishing returns.</p><p>This combination idea was first explored by MIT professors Josh Tenenbaum and Vikash Mansingkha earlier this year. Their 2023 paper, “<a href="https://arxiv.org/pdf/2306.12672.pdf">From Word Models to World Models: A Probabilistic Language of Thought</a>”, introduces a concept called “Probabilistic Language of Thought” (PLoT), a framework to combine language translation from LLMs with commonsense reasoning from probabilistic programs.</p><p>The main proposal made is a term called <em>rational meaning construction. </em>Without going into too much detail, it comprises of two parts: 1) a PLoT language that constructs structured models of “arbitrary situations” and supports language inferences, and 2) a so-called “meaning function” that takes natural language data and creates semantic meaning. The authors hope that part 1) can be accomplished by probabilistic programming, and part 2) by a pre-trained Large Language Model. The basic idea is simple: the authors’ meaning-construction model works by first using a pre-trained LLM to take raw natural language from the world and translate it into probabilistic-programming language statements. Next, the authors’ so-called “inference function”<em> </em>takes those statements and computes probabilities and inferences on those statements over possible worlds, using the classical symbolic-AI paradigm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*G7MbJXSlIiHSNn8JpkJ-3A.jpeg" /><figcaption>Citation: “<a href="https://arxiv.org/pdf/2306.12672.pdf">From Word Models to World Models: A Probabilistic Language of Thought</a>”</figcaption></figure><p>The authors give an example of their pipeline using a natural-language conversation about the game tug-of-war, as shown in the image above. First the authors use a probabilistic symbolic programming language called <em>Church </em>(visually, it looks similar to Scheme which is interesting!) to create a generative program of a tug-of-war game scenario with specific player variables (names), such as “Josh”, “Sue”, “Tom”. Once the model is created, the LLM (specifically, few-shot models) would translate a query, such as “How strong is Josh?” into Church syntax — which syntax would be fed into the generative model to create inference and output the result. An example of a natural-language query they give is: “What are the odds of Gabe beating Josh?”; and the equivalent Church-query is: (query (won-against ‘(gabe) ‘(josh))) . Notice however, that the LLM isn’t actually used as inference — the probabilistic program is used to infer semantic meaning and make decisions, <em>not</em> the LLM. It’s only valuable as an intermediary step. (<a href="https://www.youtube.com/watch?v=8j2S7BRRWus&amp;t=28s">Vikash Mansigkha gave an amazing talk</a> at TEDxMIT about this. Scroll to timestamp 9:40 for a preliminary live example of the probabilistic language of thought).</p><h3>Final Remarks</h3><p>I wish I could be a fortune-teller and predict the future of natural-language AI, but I’m not. However, there is one thing that seems to be panning out; ever-larger models will probably not lead us to human-like AI and data efficiency. With the kind of linguistic/semantic mistakes ChatGPT and earlier GPT models have been reported to make, it’s natural to wonder if we have truly reached human-like language AI. Furthermore, models such as ChatGPT and GPT-3 are data-hungry behemoths, and even steering away from LLMs for a second — the newest text-to-image generative model, Stable Diffusion (which seems to have human-like inference capabilities!) was trained on 2.3 <em>billion</em> image-to-text pairs from the <a href="https://laion.ai/blog/laion-5b/">LAION dataset</a>. And don’t be fooled — Stable Diffusion also generates images in a similar fashion to ChatGPT with text — using surface-level statistics of text-to-image structure from the training data.</p><p>A human brain would not need that much data from LLMs, relatively speaking; we’re only limited by our ability to draw (generate), which is more of an artistic skill. We can still <em>imagine </em>in our heads what text describes, even if we can’t draw it photo-realistically ; and doing that without insane amounts of data is only possible because of abstraction and high-level symbolic generation in our brain. A similar issue belies LLMs; generating a sentence in order word-by-word is very different from having a semantic idea, knowing the subject and verbs/predicates, and constructing a sentence hierarchically. One requires lots of sentence data and calculation of specific sequential probabilities, while the other requires a model with hierarchical abstraction, but potentially less data. One is able to generate the surface statistics of natural language, while another could actually infer the underlying generative process of sentence and language construction.</p><p>We’re not anywhere close to the latter, at least for now. But perhaps, combining LLMs with symbolic probabilistic programs might be the key breakthrough we need.</p><h4>A Message from AI Mind</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*5Wm7sOfTpe5DEbhg.gif" /></figure><p>Thanks for being a part of our community! Before you go:</p><ul><li>👏 Clap for the story and follow the author 👉</li><li>📰 View more content in the <a href="https://pub.aimind.so/">AI Mind Publication</a></li><li>🧠 Improve your <a href="https://www.aimind.so/prompt-generator?utm_source=pub&amp;utm_medium=message">AI prompts effortlessly and FREE</a></li><li><strong>🧰 Discover </strong><a href="https://www.aimind.so/?utm_source=pub&amp;utm_medium=message"><strong>Intuitive AI Tools</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=909367e5dbee" width="1" height="1" alt=""><hr><p><a href="https://pub.aimind.so/are-large-language-models-too-large-909367e5dbee">Are Large Language Models Too Large?</a> was originally published in <a href="https://pub.aimind.so/">AI Mind</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[p-values: A Legacy of “Scientific Racism”]]></title>
            <link>https://medium.com/data-science/p-values-a-legacy-of-scientific-racism-d906f6349fc7?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d906f6349fc7</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[diversity-and-inclusion]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[ethics]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Thu, 26 May 2022 17:12:45 GMT</pubDate>
            <atom:updated>2022-06-01T01:12:28.350Z</atom:updated>
            <content:encoded><![CDATA[<h4>Legacies of Statistics &amp; AI</h4><h4>A deeper look at the untold history of p-values and its legacy</h4><p><em>Disclaimer: I’ll be putting quotes around certain words, like race, racial measurements”, etc. because not only are these terms loaded with historical discrimination, but “race” is un-scientific. A “race” of people is an </em><strong><em>un-scientific</em></strong><em> term. Also, the reader should have a bit of familiarity with p-values, hypothesis testing, and Bayes’ theorem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z-upQVEUCTRlwyhW" /><figcaption>Photo by<a href="https://unsplash.com/@edge2edgemedia?utm_source=medium&amp;utm_medium=referral"> Edge2Edge Media</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It’s easy to think of science as objective. Science, after all, is about studying and recording what occurs in the natural world, right? The natural world is “beyond” the human realm, beyond our imagining and our minds, as science classes would teach.</p><p>But we should not get lost on <em>who</em> is doing the science: sure, a microscope will help you examine smaller objects and organisms more closely, and yes, using an X-ray telescope can display certain characteristics of a galaxy that might not be obvious in visible light — but in the end, we humans<em> </em>are doing the observing. And <em>we</em> are doing the <em>interpreting</em> of the data.</p><p>Before we delve into the history of p-values, and possible alternative solutions to the value (and their shortcomings as well), let’s briefly go over the definition of it.</p><p>In statistics, the definition is: The p-value of a hypothesis test is the probability of your test statistic taking the observed value or more extreme, <em>assuming</em> your null hypothesis is true. In mathematical notation, this would be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/0*lHUb5otTVSDOTobY" /></figure><p>The vertical line means “given H0”, which means “assuming the null hypothesis is true”. The value <em>x</em> would be, say a “z-score”, or a “t-statistic”, or a “chi-squared statistic”— if those are familiar words from your statistics classes.</p><p>One of the most common mistakes is assuming that the p-value tells you the probability of your null hypothesis given the data-evidence. This is <strong><em>wrong</em></strong>. The mathematical notation for this would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/0*AidyqX1Kh0S154Nb" /></figure><p>These two formulas are different<em> — </em>but it’s easy to confuse the two. Usually, after doing experiments, scientists want the second formula but often get stuck with the first. Later we’ll see how we could get from one to the other, and how that connects to alternative ways to measure statistical results of experiments.</p><p>Wikipedia has a nice illustration of this common error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0UzTDVF0WrNYLX_K" /><figcaption>Image source: <a href="https://en.wikipedia.org/wiki/P-value#/media/File:P-value_in_statistical_significance_testing.svg">https://en.wikipedia.org/wiki/P-value#/media/File:P-value_in_statistical_significance_testing.svg</a></figcaption></figure><p>Statisticians have a love-hate relationship with p-values, and recently have been critiquing them as a measurement of “significance” in statistical testing, and warning scientists about common pitfalls. Don’t believe me?<a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4394578/#:~:text=The%20smaller%20the%20%E2%80%9CP%E2%80%9D%20value,for%20a%20given%20sample%20size."> Read this 2015 article by the National Institute of Health</a> about it. We’ll revisit this hate-love relationship later on, so keep it in the back of your mind.</p><p><strong>What is Scientific Racism?</strong></p><p>Before I discuss the history behind the p-value, I need to briefly mention background about the phrase “scientific racism” — because not only is it loaded with years of racial policies supported by governments, but it’s one of the histories intertwined with p-values and significance testing in statistics. “Scientific racism”, or more accurately <em>psuedo-scientific racism </em>(because racism is not scientific), was a way in which European colonial governments — and the statisticians they hired to do government surveys and data collection — <em>justified</em> their racist policies by using statistical measurements, often in a extremely biased and incorrect way. For example — if you’ve ever heard of the infamous “skull measurements” used by European psuedo-scientists in the colonial era to try and demonstrate a (fake) correlation between skull size and intelligence — they did that in order to put scientific backing behind claims such as “Africans, Native Americans and Asians are less intelligent than Europeans due to smaller size.” Of course, these claims are completely un-scientific and baseless — but using scientific terminology and measurements as a backing was a way to sooth their guilty conscience.</p><p>Now that I’ve given a brief context to what scientific racism is, we’ll see more specific examples later in the piece related to significance testing and p-values. But first — a bit of history and context for the p-value.</p><p>I’ll be citing mathematician Aubrey Clayton’s<a href="https://nautil.us/how-eugenics-shaped-statistics-9365/"> wonderful article, “How Eugenics Shaped Statistics”</a> in <em>Nautilus, </em>about the history of statistics.</p><h3>The history of the P-value</h3><p>P-values didn’t just come out of the blue; they were a response to mathematicians in the 1700s and 1800s trying to understand the theoretical concept of probability &amp; randomness, and how to tie that to calculating statistical results of physical experiments — like rolling a die, flipping a coin, or more complex physics experiments.</p><p>If you’ve ever taken a statistics class (or are a statistician, or major in college) I’m sure you’ve heard of names like Galton, Pearson and Fisher. WE’ll remind ourselves that they were human beings (sometimes it was hard for me to remember that after hearing [insert name]’s formula so many times in class!). Although these individuals propped up statistics as an infallible, objective way to draw conclusions from data, even in their own work, these statisticians “revealed how <em>thin</em> the <em>myth</em> of objectivity always was”, as Clayton writes.</p><p>Galton came from a Victorian high-class family, and he often published ideas about how human intelligence could be “refined” by selective breeding in humans — by breeding the wealthiest, “learned” men as opposed to other classes in British society. Pretty unsurprising…I know. He was also alive during the 1850s, where Europe was ever-expanding in its colonialism of Asian and Africa, and America was still deeply entrenched in racist slavery and segregation. So it’s no surprise that Galton applied his “breeding” ideas to different “races” (as racist scientists used the term) of people.</p><p>But how does this tie in with <em>racism</em>? Well — when colonial European “scientists” began measuring the heights, weights and appearances of “races” in the world, they leaned on European statisticians like Galton to make conclusions based on the data. He and his contemporaries <em>believed</em> that the measurements of people in each of their “races” would follow a bell curve — the normal distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mXsSpLXxOAYhBOxC" /><figcaption>Image source:<a href="https://en.wikipedia.org/wiki/Normal_distribution#/media/File:Normal_Distribution_PDF.svg"> https://en.wikipedia.org/wiki/Normal_distribution#/media/File:Normal_Distribution_PDF.svg</a></figcaption></figure><p>If these “racial measurements” followed a normal distribution — well, since every normal curve has a specific mean and standard deviation — it “follows” that each “race” of people has an “average look”. If this logic already sounds creepy to you — that’s because it IS — and it’s frankly mind-blowing how some of these statisticians convinced themselves their research was wholly “scientific.”</p><p><em>Raveena, what about the p-value is related to this?!</em></p><p>Well, later in to the 19th century, scientists studying all manner of (non-human) biological creatures, noticed that there were quite a many “unusual measurements’’ as Clayton notes, and one statistician in the 1870s — Karl Pearson — yes, of beloved Pearson’s correlation — wondered how to decide if data points were normally distributed. In theory, if you could figure that out, that would tell you a boatload of information, especially on detecting outlier (“unusual”) data that scientists were pining over.</p><p>As Clayton notes, before Pearson, statisticians would simply draw a histogram of the results, and then basically play connect-the-dots (or connect-the-bars??) to see if it looked like a bell curve. But Pearson got more mathematical in his approach, and created a 3-step process:</p><ol><li>Assume the data fits some type of distribution you initially had in mind. This is the “null hypothesis” I mentioned at the beginning of the article. Also write down your “alternative” hypothesis.</li><li>Compare the data to what you expect from the null-hypothesis assumed distribution of data, and calculate what’s called Pearson’s “chi-squared test statistic.”</li><li>Compute the p-value of this test statistic, and determine if the result is “significant”. And assess if this provides any evidence in favor of the alternative hypothesis.</li></ol><p>Clayton notes how Pearson didn’t actually believe that statistical significance necessarily meant “”worthy of scientific discovery” — he literally just used the word to mean, “oh, the data is indicative of a different hypothesis.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S7H7xVSClpHijwZJ" /></figure><p>So here, we can see that the “outliers”, or “unusual observations” are the un-likely ones marked off by the test statistic. And the p-value is just the total area (probability) of that region.</p><p><strong>The Cardinal Sin of Justifying Racism with Statistics</strong></p><p>But the main sin that Pearson and his statistician contemporaries committed was not applying these tests to crabs, as Clayton points out — but to <em>human beings</em>. Humans are way too diverse in our life experiences and cultures, but that didn’t stop Pearson from trying to apply his statistical methods to “biological measurements” made on different “races” of humans, as he thought of it. In the early 1910s Pearson did a statistical study of 4,000 pairs of siblings in the UK and “determined” that statistical measurements of eye color and other body regions had “strong correlations” — according to the p-value significance test — with their character, such as “assertiveness and introspection” (Clayton). By today’s standards of science we would immediately dismiss this claim, but just a decade later, when Jewish refugees fled to Britain escaping Eastern Europe — Pearson applied the <em>same </em>flawed p-value correlation analysis to<em> Jewish children — </em>and he concluded from his significance testing that they were of “inferior stock” and their “intelligence had almost no correlation with environmental factors and discrimination.” (Clayton) If you think you’d be laughing in his face for this judgment — you’re not alone — but don’t forget that Pearson’s correlation still exists in Python and R libraries. It’s been very difficult to separate the fact that these statisticians supported eugenics judgements using mathematics, from their very valid impressive contribution to the field.</p><p>Statisticians like Pearson, Galton, and later professors like Davenport believed in eugenics, racism, and would use statistical arguments to back up horrible theories on the “dangers of interracial mixing”, “breeding within the race to preserve intelligence”, and various other arguments (Clayton). Pearson even involved himself with “eugenics departments” and societies in universities and outside in America — organizations which were adamant about preventing race-mixing between White Americans and people of color (Asians, Africans, Native peoples, etc)</p><p><strong>So What’s the Story Now?</strong></p><p>Fast forwarding 100 years later — in 2019, almost a thousand statisticians and scientists called for a deep critique of significance-testing, and their call for change was echoed at the American Statistical Association. In fact, the statistician Ron Wasserstein gave an amazing talk <em>in that same year</em> at the United States Conference at Teaching Statistics on why the p-value, especially the low-bar for significance at p &lt; 0.05, <a href="https://www.causeweb.org/cause/uscots/uscots19/keynote/2">should be re-evaluated</a>.</p><p>Their valid arguments point out that making a significance decision based on two hypotheses (which seem like an arbitrary number) aren’t that meaningful of a statistical task, and given enough variables in your data, “spurious correlations” — as they are called — can often occur, and just due to the multiplicative and complementary nature of probability, the chances of at least one “significant measurement” by standards of the “p &lt; 0.01” criteria will happen — even if that measurement is <em>not meaningful</em> to the observer.</p><p>So if p-values and as it’s called “classical hypothesis testing” is in need of critique and review, what are some alternatives, and are they the “best” options out there?</p><h3>Possible Alternatives</h3><p>So if p-values and significance testing are flawed and in need of deep critical analysis, what are alternatives to this? Are they good alternatives? Do they visit similar vague and risky territory as p-value significance.Well, the answer is, as always: it’s complicated, but there are some avenues.</p><p><strong>Prior Context to Statistical Data</strong></p><p>In the beginning of the article, I mentioned how it’s easy to think of statistics and scientific data measurements as objective, when they really aren’t — the key point is data doesn’t speak for itself — it must be <em>interpreted </em>in a<em> larger context </em>to reach an actual conclusion.</p><p>For example: say I find a coin off of the sidewalk and do an experiment: I flip it 10 times, and it comes up with 9 heads, 1 tail. Now say person A, Jason, had never seen a coin before — maybe they’re a time-traveler from the year 2100 where no coins exist, their society has no historical memory of coins, and they only use electronic payment — just go with me here! How would Jason interpret the data? Well, the data suggest that since 9 out of 10 flips are heads, the probability of a head “should” be 9/10. At least, the evidence is <em>consistent </em>with the hypothesis that p = 9/10. Since Jason has never seen a coin before, it makes sense that he would believe just the evidence.</p><p>Person B, Verity — is from 2022, and definitely has seen coins and held them before and has common knowledge about them. Again, the data suggest that since 9 out of 10 flips are heads, the evidence is consistent with the hypothesis that p = 9/10. BUT, here’s the catch for Verity: From Verity’s experience (and society), randomly-found coins on sidewalks usually are 50–50 fair, so she has <strong>prior context </strong>about how coins work. She would hold the “fair coin” (p = 1/2) assumption pretty highly in their mind and conclude that the 9-heads data was a statistical fluke in the context of their hypothesis.</p><p>(By the way sorry, Jason — I get that you come from a vastly more technologically advanced time period than me and Verity, but…this time around, I was flipping a fair coin. The lack of context about coins skewed your conclusion!)</p><p>If I had flipped the coin 100 times, and I got say 90 heads — then that might be enough data for Verity to change her assumption about the coin being fair — in other words, the odds of her believing the coin are biased go up. MIT cognitive scientist Joshua Tenenbaum<a href="https://www.youtube.com/watch?v=icEdI0AIOlU"> has a great Youtube video</a> detailing this exact experiment, around the 37:20 mark.</p><p><strong>Interpreting the Data to Make Conclusions</strong></p><p>Notice earlier when I mentioned Verity — she interpreted the same data <em>in a different context</em> of her hypothesis. The word “context” is important. It turns out that the field of “Bayesian statistics”<em> </em>is quite concerned about the contexts of hypotheses — what it calls “prior context”, or “prior assumptions” — and the prior assumptions Jason and Verity made greatly skewed the conclusions they made <em>on the same data</em>. And turns out, Bayesians are trying to provide alternatives to the usual significance testing.</p><p>One of these alternatives is called “Bayes factors”. The Bayes factor, in a sense, helps compare whether data under the null hypothesis is <em>more likely</em> or not than the alternative one. It basically tells us how much the data (evidence) support the alternative over the null, which can be very useful. Mathematically, it’s written like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/0*blmS191FPiosWbgU" /><figcaption>Image credit: Author. Equation 1 is the Bayes Factor, written as “K”. Eq. 2 is Bayesian inference, written as odds.</figcaption></figure><p>If you’ve ever seen bettors write their gambling odds as “2-to-1”, or “the odds of a win like that is 1 in 1,000”, that’s the same type of odds we are using here. “2-to-1” and “1 in 1,000” as odds would be written as 2:1, and 1:1000, respectively.</p><p>Steven Goodman, a Professor of Medicine at Stanford University,<a href="http://www.statlit.org/pdf/2001-Goodman-Epidemiology.pdf"> has a great article here</a> comparing Bayes Factors and P-values and notes that Bayes factors around 200 — which means that the alternative hypothesis is 200<em>times more likely</em> to be true than the null — often have a “corresponding” p-value around 0.001, indicating that “classical significance” might need to have stronger p-value conditions. Statistician Brendan Kline of the University of Texas, Austin<a href="http://www.brendankline.com/papers/Kline-Bayes-Sept2020.pdf"> notes in his research paper</a> that Bayes factors around 0.1 denote “strong” evidence against null hypothesis in the context of <em>big data</em>, and for hypothesis testing of hose types of data sets, a corresponding p-value for “significance should be less than 0.0025” (Kline). And back in 2018,<a href="https://bmcpsychiatry.biomedcentral.com/articles/10.1186/s12888-018-1761-4#Sec9"> this article in <em>Springer Nature Psychiatry</em></a> mentions that the strengths of Bayes factors are complementing p-values and effect sizes when it comes to hypothesis testing. However, the strength of the Bayes factor does depend on the “width of the prior distribution<em>”</em>, which is set by the researcher — so again, if you’re looking for some “objective” measure you’re out of luck. Finally, the Bayes factor <em>by itself</em> does not determine which hypothesis you, the researcher, should choose — you must combine this with the prior odds ratios determined wholly by the researcher before seeing any data.</p><h3>Final Thoughts</h3><p>I hope by reading this, you’ve gotten a sense of the history of p-values, problematic applications of p-values and classical hypothesis testing, which revealed its limitations, and the attempts researchers are taking to remedy those challenges. I don’t want readers to think that p-values are useless after reading this article. P-values are much more common in statistical software like R, or Python’s <em>scikit-learn</em> &amp; <em>scipy.stats</em> libraries — Bayesian techniques are very new to Python’s data libraries. But it’s important to understand <em>why</em> the critique on classical testing exists, and an underlying theme of statistics is that subjectivity is <em>everywhere,</em> through and through in the field. The data almost never speak for themselves. It has to be interpreted in the surrounding context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d906f6349fc7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/p-values-a-legacy-of-scientific-racism-d906f6349fc7">p-values: A Legacy of “Scientific Racism”</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Revisiting MNIST with Fresh Eyes]]></title>
            <link>https://medium.com/data-science/revisiting-mnist-with-fresh-eyes-36f9d19a75d1?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/36f9d19a75d1</guid>
            <category><![CDATA[gaussian-mixture-model]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[mnist]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Mon, 21 Mar 2022 23:36:32 GMT</pubDate>
            <atom:updated>2022-03-27T05:10:48.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hands-On Tutorial; Shifting Perspective in AI</h4><h4>A spatial perspective to MNIST data, rather than the usual vector-treatment</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GtoBopXUw3Grf9TO" /><figcaption>Photo by <a href="https://unsplash.com/@enka80?utm_source=medium&amp;utm_medium=referral">nine koepfer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>By 2022, the MNIST dataset has become the “Hello-World” dataset of people entering AI. Hell, it’s probably been that for a few years already.</p><p>In the usual AI treatment of MNIST––whether loading the data into a neural network, logistic regression, random forest, insert-AI-algorithm-here, each handwritten digit image, with its pixel values, is treated as a <em>vector</em> (so the square image is elongated into a vector), and the vector is fed into the algorithm. For thousands of digits, training the algorithm means feeding in thousands of these pixel-elongated vectors as a matrix into the algorithm to crunch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pglEhPlTN9yjBM2BGxHJ-A.jpeg" /><figcaption>Image credit: <a href="https://medium.com/dataman-in-ai/module-6-image-recognition-for-insurance-claim-handling-part-i-a338d16c9de0">https://medium.com/dataman-in-ai/module-6-image-recognition-for-insurance-claim-handling-part-i-a338d16c9de0</a></figcaption></figure><p>However, I want you, the reader well-versed with AI, to take a different perspective to the digits — one that you almost take for granted with your own human vision since it seems quite intuitive––and our eyes are quite literally built for this perspective.</p><p>Image data has a unique trait that other types of data, such as tabular records, NLP-text data, audio data, etc. don’t have–the intuitive <em>spatial </em>configuration<em> </em>of the pixels in the image. So instead of trying to remove the spatial property of image data by coalescing the rows into a single vector, let’s take advantage of this property that our eyes &amp; brain use intuitively!</p><p><strong>Converting Image Pixels into Cartesian Points</strong></p><p>The first step we need to take is to convert the image pixels into cartesian (x,y) points. We take a single image, and transform the values, so we can make a copy of it using dots in the <em>xy</em>-plane. So this is the original pixelated image of a 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*KojuWCJgU_50R4josbwSuw.jpeg" /><figcaption>Image Credit: Author. The original sample MNIST image of a “3”</figcaption></figure><p>Each pixel in the black-white image above has a coordinate: (0,27) is the bottom-left and (0,0) is the top-left. So it’s a bit different from the usual Cartesian <em>xy-</em>plane where (0,0) is in the <em>bottom-left</em>. First we do a transformation to convert pixel-coordinates to Cartesian coordinates, which is done in this code below:</p><pre>normalize_image = np.asarray(image).astype(&#39;float32&#39;)/255.0<br>    coordinates = list(itertools.product( range(0,28) , range(0,28)))<br>    ...<br>    x_values = np.float16(np.asarray([fp[1] for fp in filtered_points]))<br>    y_values = 27-np.float16(np.asarray([fp[0] for fp in filtered_points]))</pre><p>Then, we add statistical <em>noise </em>(using a normal distribution) to each point, so that it “fills-up” the space more between the pixel-points. Each pixel is at an integer-coordinate; so converting to Cartesian only gives us integer coordinates, which isn’t a data-rich representation. The basic idea is that we go to each integer Cartesian coordinate–converted from pixels–and sample 4 new points as random noise <em>around</em> the selected coordinate, and plot all 5 points together. The code is this:</p><pre>for i in range(0, data.shape[0]): # this loops thru each Cartesian coordinate<br>      point = data[i,:]<br>      mean = point<br>      cov = np.array( [[0.0001, 0], [0, 0.0001]] )<br>      ...<br>      test_x, test_y = np.random.multivariate_normal(mean, cov, 4).T<br>        ... # this creates the random noise<br>    return filled_in</pre><p>And the finished Cartesian, coordinate <em>xy-</em>plane result looks like this!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*LmXAgW9gZHdUkvvzGKWVNQ.jpeg" /><figcaption>Image Credit: Author. You can see here that this image made from the Cartesian pixel-integer coordinates (normalized), added with noise, looks really similar to the MNIST black-and-white image above!</figcaption></figure><p><strong>Here Comes the Machine Learning!</strong></p><p>Okay, so you’re probably asking: why did we go through this whole process just to make <em>another</em> look-alike of the MNIST image with Cartesian points? What’s the big deal?</p><p>Well, the big deal is: because we now have (x,y) points as data, we can perform machine learning on this as if it were any other 2D dataset! The machine learning technique we’re going to use is called Gaussian Mixture models. Basically, it’s a fancy way of teaching the computer ways to understand the distribution of points, by using Gaussian <em>densities</em> (just think of it as Gaussian ovals; they literally look like ovals, as you’re about to see in a second).</p><p>The code to create this is: (“plot_gmm” is a custom visualization function, credit goes to Jake VanderPlas for the code to visualize the clusters. His function can be found here: <a href="https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html">https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html</a>)</p><pre>selected_model = GaussianMixture(n_components=gaussian_learning( result ), random_state=1, covariance_type=&#39;full&#39;)</pre><pre># plot the selected model over original image data with a user-defined function. Credit to Jake VanderPlas.<br>plot_gmm(gmm=selected_model, X=result)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*QJSgcyTlmzxOSup1E8jTUA.jpeg" /><figcaption>Image credit: Author</figcaption></figure><p>Nice colors! But what does this tell us, concretely? Well, it shows us that the AI algorithm was able to figure out the mixture pieces required to create points in the shape of that “3”.</p><p>But wait a minute: this was for only one specific “3” in the dataset. Wouldn’t we have to do this visualization for every single “3” in the MNIST set? Well, yes, but instead of doing a visualization, let’s focus on a different property gleaned from this set-up: the <em>number</em> of mixture <em>components</em> used in the mixture machine-learning process.</p><p><strong>Aggregating Interesting Properties Across the Data</strong></p><p>Let’s look at the image above again. We can see that the ML algorithm used 6 mixture components to derive the distribution of points that make a “3”. Is this common across all “3”s in the set? And why 6 components? To our human eyes, it just looks like “3” is composed of two simple curves joined at the center.</p><p>To answer the second question–why 6 components– in the visualization above, we see a limitation to using the Gaussian cluster: each component is an an ellipse with imaginary perpendicular axes, kind of like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*7B72CpkZ9rw52CoB90MDvA.png" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Ellipse">https://en.wikipedia.org/wiki/Ellipse</a></figcaption></figure><p>Why is this important? Well, an ellipse with these perpendicular axes can certainly cover a “cloud” of points vaguely distributed like ovals. But this puts a restriction: in the above image of data-points shaped like a 3, the main sections of the points which comprise the “curves of the 3” look like convex curved “jellybeans”–and while an ellipse is also convex, it can’t be forcibly curved or bent like a jellybean without changing the equation. And changing the equation, means not using Gaussian mixture learning.</p><p>So this basically tells us that Gaussian mixture modeling, in the context of MNIST from this 2D, spatial configuration–can really only model “straight pieces” of the graph. It’s almost as if you were tasked with taking the picture of a 3, and breaking it down into polygon straight sections, and then drawing ovals around those straight lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*YdPxAnL65MH1CEnscR6uzA.jpeg" /><figcaption>Image Credit: Author. Breaking a “3” piecewise into linear pieces, then envelop each straight line with a oval. Looks similar to the Gaussian colored clustering above, doesn’t it?</figcaption></figure><p>So imagine that all the data points from the visualization above that composed the 3, each of them belonged to one of these ovals. As you can see, the straight lines here are the major-axis diameter of the ellipse, and depending on the sizes of the ellipse and the data-point configuration, there might be more or fewer ellipses. But Gaussian mixture learning is attempting to dissect the image into these clusters, which explains why some “3”s are clustered with 6 components.</p><p>But obviously in the illustration I’ve drawn more than 6 ellipse components, so which diagrams require more or fewer than 6? To do that, we need to visualize a histogram of the # of components. I won’t show all the code here, but for this example we’ll be taking a random sample of 300 images out of 6,000 (so sampling 5%) of a “3” from the MNIST dataset, feeding them through the Gaussian Learning process, and visualizing the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*pLNcbLZXoQ_lkFs6JzYc1A.jpeg" /><figcaption>Image Credit: Author</figcaption></figure><p>According to this chart–and extrapolating to the whole MNIST dataset based on the <em>sampling normality assumption </em>— between 37% and 43% of “3”s require six components — six piecewise clusters to create the image of a 3. Meanwhile, between 24% and 30% of “3”s would require five components, and between 15% and 20% of “3”s would require seven components.</p><p>So…what can we glean from this information? Let’s look back at the visual illustration of the dots. If the “3” is drawn quite small, compact, then it wouldn’t require as many piecewise straight lines to compose the image — and hence, five or six Gaussian components should be enough. But if the “3” is drawn much larger, or say with a loop in the middle which sometimes happens, then the drawing would require more components to cover. According to the statistics of the histogram, the average # of components is around 6, and the standard deviation of the sample is 1-component, so it’s not hard to see an example where seven components are necessary. In fact, here’s an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*zBOOCJSILkTTz0Cq4WwD0Q.jpeg" /><figcaption>Image Credit: Author. You can count that there are 7 components here, because the person drew an extra curl on the left-bottom end of the 3, and an emphasized bend in the middle.</figcaption></figure><p><strong>Final Thoughts</strong></p><p>This isn’t, by any means, necessarily a path to a new machine-learning algorithm for MNIST images. The MNIST image has been trained like an old dog for <em>ages</em> on AI algorithms. Rather, what I want to present to the reader is an alternative, <em>spatial</em>-inspired perspective of applying machine learning techniques to handwritten digits.</p><p>This Gaussian Mixture Learning process won’t necessarily gain better results over a neural network; but the goal here isn’t results, it’s simply a change in perspective. Taking advantage of the spatial positioning of pixels, and translation into data points, allows us to use the visual geometry of Gaussian clustering to look at, what basically is– a <em>geometrical</em> result–a handwritten digit. It also allows us to use concepts even from high school and college geometry!–the idea of <em>piecewise connecting</em> dots on a curve using straight lines, treating it almost as sides of a polygon. This concept isn’t new; in fact, piecewise connections are used in calculus to find the “arc-length” of curves! But here, intuitively reasoning about piecewise lines geometrically, actually gives us good reasoning behind the internal structure of the mixture model–and explains why some handwritten drawings of the same digit use more pieces (components) than others.</p><p>Thanks so much for reading, and stay tuned for my next article! In the meantime, feel free to check out <a href="http://www.medium.com/@nonbinarycat97">some of my other past articles</a> on my thoughts of AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36f9d19a75d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/revisiting-mnist-with-fresh-eyes-36f9d19a75d1">Revisiting MNIST with Fresh Eyes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prior Knowledge in AI — Is it Really “Cheating”?]]></title>
            <link>https://medium.com/data-science/prior-knowledge-in-ai-is-it-really-cheating-229604179e7f?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/229604179e7f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[human-brain]]></category>
            <category><![CDATA[data-justice]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Thu, 27 Jan 2022 18:43:09 GMT</pubDate>
            <atom:updated>2024-02-27T17:20:35.060Z</atom:updated>
            <content:encoded><![CDATA[<h3>Prior Knowledge in AI — Is it Really “Cheating”?</h3><h4>Why prior knowledge is so important for AI systems</h4><p>Let’s start things off with an optical illusion: the famous grey checkerboard illusion, often shown as an example of optical illusion that fool our brain’s and visual processing system. I want to focus on a specific illusion of the grey checkerboard that’s quite popular involving <em>color</em> and <em>illumination</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHa91hhUedXq_TU--VT3cQ.jpeg" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Checker_shadow_illusion#/media/File:Checker_shadow_illusion.svg">https://en.wikipedia.org/wiki/Checker_shadow_illusion#/media/File:Checker_shadow_illusion.svg</a></figcaption></figure><p>I want to ask you a simple question: is square A the same <em>hue </em>as square B? Clearly not! Square A and B are definitely gray, but A obviously has a darker hue than B…right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LB9gKMG4ufpO1AUiMU489A.jpeg" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Checker_shadow_illusion#/media/File:Grey_square_optical_illusion_proof2.svg">https://en.wikipedia.org/wiki/Checker_shadow_illusion#/media/File:Grey_square_optical_illusion_proof2.svg</a></figcaption></figure><p>Wait a second….removing the shadow by connecting the squares with a gray bridge…why are they appearing as the same hue now? What the heck is going on?!</p><h4>How Does the Brain Reach its Conclusions?</h4><p>Why were our brains so quick to perceive the two hues as different, while removing the shadow revealed that the hue intensity values are actually the same? It comes down to understanding how our brains reached the primary conclusion — the way our brains <em>inferred</em> its conclusion.</p><p>The hope for AGI — artificial <em>general</em> intelligence — at least its hope in the 60s and 70s, and perhaps now too — is to try and emulate human intelligence, and the powerful extrapolation and generalization and inference capabilities our brain has, and combine that with the extraordinary computational powers of computers. So we should try to understand first how our brains might have reached its primary inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f25FTonsjmyVobe5" /><figcaption>Photo by <a href="https://unsplash.com/@miracleday?utm_source=medium&amp;utm_medium=referral">Elena Mozhvilo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In order to understand this illusion, we need to appreciate that when lookung at the checkerboard illusion, our brain was judging 2 hypotheses simultaneously and unconsciously:</p><p><em>H1: Square B is really a different hue than Square A –– just viewed in the shadow</em></p><p><em>H2: Square B is the same hue than Square A––just viewed in the light</em></p><p>Now, let’s think about the variable <em>D</em>, as data — in this case, the “data” here is the pixel color-intensity values of both center squares arriving to our eyes and registered there. It turns out that our eyes kinda subconsciously “knew” that the pixel intensities of the two squares were the same! The data is <em>equally consistent</em> with both hypothesis H1 and H2 according to your eyes. But your eyes are not the full story––ultimately your <em>brain</em> has to make sense of those measurements. And this is where the illusion happens.</p><p>Take a look at both pictures above: do you notice what’s different with them? It isn’t the checkerboard, but what surrounds the back of it — the cylinder’s <em>shadow</em>. Your brain has <em>prior knowledge </em>(from daily experience, or perhaps innately from evolution)<em> </em>for how shadows are created by light, and how shadows affect colors and darken their apparent hue.</p><p>If our brains go along with this general shadow hypothesis assumption, the only way that Square B on the frontal face could actually be the same gray hue as Square A, is if <em>only</em> Square B was illuminated in light — a single skinny light source surrounded by an object’s shadow. Has your brain ever seen examples of shadows “surrounding” a skinny light source from past experiences? Probably…..not.</p><p>So in the end, your brain judges H1 to be true — that Square B is really a different hue than Square A — because our brains incorporate prior 3-dimensional context for shadows into the final decision.</p><p><strong>But wait! Both colors are still the same hue of gray!</strong></p><p>Yes, they are — and that’s what makes this an <em>illusion.</em></p><p>Without this prior knowledge about how shadows are created and affect colors, both hypotheses H1 and<em> </em>H2 are equally valid, and so your brain would be confused all the time because it wouldn’t be able to distinguish between the two and decide on a particular hypothesis. That’s why the illusion happens; your eyes unconsciously process H1 and H2 as equally valid, but your brain unconsciously adds in the prior knowledge. Only after the big reveal in the second picture , do you consciously get confused by the shadows being masked to reveal the hue intensities.</p><p><strong>So what? Do our brains just lie to us all the time? Is everything an illusion?</strong></p><p>Probably, yeah.</p><p>In fact, this is what Prof. Olshausen, a computational neuroscientist of the Simons Institute for Neuroscience argues. By the way, <a href="https://www.youtube.com/watch?v=PLAZyu73tWE&amp;t=874s">in this brilliant short video</a> he walks the viewer through a very similar illusion but with the famous Rubiks’ Cube, and explains how our brains infer the colors based on combining what he calls a “general shadow prior” with our eyes’ visual measurements in an elegant inference. The point is, our brains perceptions are often subjective, and depend on the priors we have built up over experience, as well as how our brains <em>combine</em> the priors with the eyes’ visual processing. Different people have different personal experiences, and some people’s inference mechanism works in different ways than others.</p><h4><strong>OK, but how does this relate to AI? That’s what you put in the title. So far, you’ve just told me about the brain! Meat ≠ Electronics.</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1GdSXfE6Jqf7_mRZ" /><figcaption>Photo by <a href="https://unsplash.com/@mediamodifier?utm_source=medium&amp;utm_medium=referral">Mediamodifier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The type of probabilistic inference your brain used to infer the color or shape of an object is similar to Bayesian inference — and it’s possible the brain approximates this type of calculation and inference at different levels of abstraction when modeling the world and making decisions, actions and planning.</p><p>“Bayesian inference”, or its computational cousin — “probabilistic programming” — might sound like new concepts because both are getting more press coverage in the data science and AI world. But probabilistic inference has been a key player since the birth of AI nearly 80 years ago. In fact, Alan Turing — one of the originators of AI — wrote extensively on probabilistic inference back in the 1940s and ‘50s. Turing’s assistant during World War 2, I.J. Good, <a href="https://www.jstor.org/stable/2335677">wrote a 1979 biography</a> of his experiences with Turing, and mentions that he “<em>directly introduced</em> the idea of the Bayes Factor” and was one of the “independent inventors of sequential analysis [time series] using Bayesian inference”.</p><p><strong>So what’s with the title of your article about prior knowledge and “cheating”?</strong></p><p>Well, unlike Alan Turing’s days in the 1950s, 21st century AI is dominated by deep learning — which, by the way, refers to the amount of layers in a neural network — not necessarily about how well the neural network can “abstract away” concepts and objects.</p><p>Let me repeat that. “Deep neural networks” do <em>not </em>necessarily learn a concept “deeply” like humans do. The word “deep” in this context is taken very, very literally.</p><p>One of the main issues with deep learning — and this also goes for logistic regression, and many other algorithms (I just focus on deep learning because it’s become the “face” of AI) — is that it’s often completely data-driven. It often doesn’t incorporate prior knowledge. These types of models, called “supervised models”, often banish the prior and try to learn object recognition, or natural-language understanding completely from scratch. Ever wonder why training deep learning models from scratch (I’m not talking about pre-trained models!) take forever on your laptop if you don’t have a special GPU chip? Because without prior knowledge about the world, systems like GPT-3 need literally <em>billions</em> of data to train and perform inference about natural language in our world.</p><p>Perhaps prior knowledge is seen as “cheating” in AI because otherwise a machine wouldn’t be “learning by examples”? After all, the original definition of machine learning was about machines learning by examples without being explicitly programmed. If you inject prior knowledge into the machine, isn’t that explicitly programming it?</p><h4><strong>Why is Prior Knowledge so important for inference in AI?</strong></h4><p>But some of the mistakes of GPT-3 — one of the “kings” of NLP AI — mistakes such as GPT-3 saying that pencils are heavier than toasters— shows that GPT-3 has <em>no prior knowledge</em> about the world. None of these AI systems understand that language ties concepts in our head to the real world. In order to infer that a toaster is heavier, you need to have prior <em>physical </em>knowledge of the world that toasters are heavier than pencils.</p><p><a href="https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html">Kevin Lacker, on his blog, gave GPT-3 a Turing Test </a>— and it performed surprisingly well on many questions! Eventually though, Kevin was able to stump the AI by asking common-sense questions that humans would never think about — because our brains have been able to infer those answers with prior knowledge and years of inference techniques evolution gave us — that’s why those questions seem “obvious” and mundane. Here’s one of those questions:</p><pre>Q: How many eyes does my foot have?<br>A: Your foot has two eyes.</pre><p>A human would <em>never</em> make that mistake, because we have prior knowledge about the world and human feet. But GPT-3 has no prior knowledge — it only knows about the websites it has been trained on — a very fancy, complicated statistical search engine. Here’s more questions Kevin tried:</p><pre>Q: How many eyes does a blade of grass have?<br>A: A blade of grass has one eye.</pre><pre>Q: How many rainbows does it take to jump from Hawaii to seventeen?<br>A: It takes two rainbows to jump from Hawaii to seventeen.</pre><p>He mentions that these type of “mundane” questions are ones you’d never see on online Web documents — because humans don’t talk about these on online documents. We learn and infer these from daily knowledge and common-sense.</p><h4>Commonsense Reasoning: Where Priors Shine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kenfUUNE6GzlRkwM" /><figcaption>Photo by <a href="https://unsplash.com/@averey?utm_source=medium&amp;utm_medium=referral">Robina Weermeijer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you still think prior knowledge in an machine-learning/AI system might seem like “cheating”, let’s look at the last question/answer pair.</p><p>Our prior knowledge is that Hawaii is a location; a piece of land. GPT-3 probably doesn’t have this type of hierarchy of abstraction in its memory. Humans also know from prior knowledge that 17 is a number, and <em>not</em> a location. And, from prior knowledge — which is derived from our <em>experience with the physical world</em> — you cannot travel from a location to a number. Not because we were taught that doesn’t work using input-output examples — but because <em>physically</em>, our priors tell us this is unreasonable. And finally, we know from prior experience that you cannot stand on a rainbow or jump on it — because a rainbow is not an object humans can interact with. GPT-3 knows <em>none</em> of this — <em>no priors</em>. It has had zero experience with the physical world to answer this type of question. It cannot answer, “ This is an unreasonable question!”</p><p>So are humans, in a sense, “pre-built” with Prior Knowledge?</p><p>In a sense, yes. For example, it’s been shown by neuroscientists that, at least in the domain of vision, our brains are loaded with shape priors that help us detect objects. It’s one of the primary reasons why <a href="https://www.quantamagazine.org/where-we-see-shapes-ai-sees-textures-20190701/">AI deep-learning systems often put more weight on <em>textures</em> rather than shapes</a> when it comes to visual recognition. A lot of our prior knowledge is derived from our brains and bodies interacting with the physical world. But AI’s cannot currently do this! The only way we could ostensibly have an AI system learn from the world is 1) create a robot to interact with our physical reality, or 2) create a <em>virtual reality</em> (similar to ours) for the AI to interact.</p><p>More interestingly, AI practitioners have recently started to inject prior knowledge into the <em>weights</em> of deep neural networks, and have seen more robust generalization as a result. <a href="https://arxiv.org/pdf/1810.05193.pdf">This study from 2018</a> mentions how putting prior-distributions on weights in a deep neural network help “regularize” the network, preventing it from overfitting too closely to the data. <a href="https://ojs.aaai.org/">Another article from 2020</a> talks about how injecting strict priors on video/audio deep-learning system helped provide reliable uncertainty estimates. And applying priors on neural-net weights has started to lead to development of a <em>foundational theory</em> <em>of deep learning</em>, detailed in <a href="https://arxiv.org/abs/1711.00165">this important research paper from 2017</a>.</p><p>I’d like to conclude with a final thought.</p><blockquote><em>Biased data will always exist, because we live in an unequal world. But if we injected our AI with prior knowledge, and gave it the ability to reason with common sense and align its goals with the goals of human equality, equity, co-operation, collective justice — those AI systems would be better equipped to interact with our world.</em></blockquote><p><strong>References</strong></p><ol><li>Good, I. J. “Studies in the History of Probability and Statistics. XXXVII A. M. Turing’s Statistical Work in World War II.” <em>Biometrika</em>, vol. 66, no. 2, [Oxford University Press, Biometrika Trust], 1979, pp. 393–96, <a href="https://doi.org/10.2307/2335677.">https://doi.org/10.2307/2335677.</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=229604179e7f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/prior-knowledge-in-ai-is-it-really-cheating-229604179e7f">Prior Knowledge in AI — Is it Really “Cheating”?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model-Based Learning on Cancer Data]]></title>
            <link>https://medium.com/data-science/model-based-decision-making-for-health-data-581bd58a1a3?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/581bd58a1a3</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[generative-model]]></category>
            <category><![CDATA[model-based-learning]]></category>
            <category><![CDATA[health-data]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Mon, 24 Jan 2022 17:34:57 GMT</pubDate>
            <atom:updated>2026-03-24T16:59:55.774Z</atom:updated>
            <content:encoded><![CDATA[<h4>Model-Based Learning; Hands-On Tutorial</h4><h4>Tutorial on deploying model-based AI on imbalanced and missing health data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1RiP6g33eWD8C3VE" /><figcaption>Photo by <a href="https://unsplash.com/@nci?utm_source=medium&amp;utm_medium=referral">National Cancer Institute</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, I’ll be giving a start-to-finish tutorial overview of applying model-based learning to real-world health data — in my case, carcinoma data. At the end of the article I hope you’ll gain a sense of why model-based learning can be a helpful general framework to apply to many different datasets, and how it can help identify the kinds of specific classification decisions to make based on data domains like healthcare.</p><p>For example, in the healthcare domain, predicting someone is healthy when in fact, they might need surgery or treatment — can dave <em>deadly</em> costs, as their underlying condition might not be seen by a human doctor until it’s too late since they’ve been deemed healthy by the automated system. Whereas, the model predicting cancer for the patient while they’re actually healthy — isn’t as bad of a thing, since the patient can just get regular checkups anyway through their insurance. And if a condition does develop their doctor will be the first to know.</p><p>The dataset I’ll be working with is <em>real </em>data from the Coimbra Hospital and University Centre in Portugal, and consists of 165 patients with “Hepatocellular Carcinoma”, a type of common but deadly liver cancer often associated with Hepatitis A &amp; B.</p><h3>Initial Data Analysis (EDA)</h3><p><strong>Fixing the Missing Health-Record Data</strong></p><p>First, we need to download the data from the UC Irvine Machine Learning Repository <a href="https://archive.ics.uci.edu/ml/datasets/HCC+Survival">at this link</a>. You’ll notice that in the link, it says that about 10% of the data is <em>missing</em>, with <em>only 8 patient records having complete data</em>. So our first step will be to fix that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YxbvMuZQ5SwI6CJJFhjY2g.jpeg" /><figcaption>Image Credit: Author. The missing values are labeled with a question mark “?”.</figcaption></figure><p>Before we do any feature selection or machine learning, we first have to fix the missing data. Usually, <em>pandas</em> library will provide some type of option, such as pandas.fillna which basically fills in the missing values in each feature column with either the <em>mean</em>, <em>median </em>or<em> </em>nearest-neighbor (if the column is numeric), or <em>mode </em>(if the column is categorical).</p><p>In this case however for our carcinoma data, the authors of the paper (who kindly published the dataset for free on UCI) outlined a missing-data scheme based on a patient similarity metric called “Heterogeneous Euclidean-Overlap Metric” (HEOM), which according to the authors works well for filling missing values in datasets with continuous <em>and</em> discrete types. The code contains details in Python on how to implement this; <a href="https://github.com/galaxyenby1997/carcinoma-modeling">you can view it here</a>.</p><p><strong>Finding the Best Features</strong></p><p>Now that we’ve filled in our missing data, let’s figure out how to pick our features. I decided to not spend too much time on feature selection. There are many ways to achieve this: using recursive feature elimination (RFE), mutual information, chi-squared tests for independence, and stuff. I wouldn’t recommend RFE usually because with enough features, the number of combinations you need to test grow very fast.</p><p>In this case, for the health records, I’m going to do something else entirely: I’ll use a random forest to determine the features. You may be asking, “why the hell would I use a random forest to figure that out??” but it turns out random forests have a built-in attribute called <em>feature_importances</em> that is determined by the collection of decision trees used. (By the way, if you would like a quick review article on Random Forests, <a href="https://towardsdatascience.com/whats-in-a-random-forest-predicting-diabetes-18f3707b6343?source=your_stories_page----------------------------------------">check out my previous post</a> on them, with an application to real-world diabetes data!)</p><p>Using a random forest, we find that the top 3 features predictive of patient survival are, in order: Alpha-Fetoprotein (ng/dL), Haemoglobin (g/dL), and Alkaline Phosphatase (U/L). (These health-record attributes are also listed on the <a href="https://archive.ics.uci.edu/ml/datasets/HCC+Survival">UCI database where this dataset</a> is found.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*FtoBfvmo624xQX-Zco4AQw.jpeg" /><figcaption>Image credit: Author. The top 3 features correspond to the highest 3 bars.</figcaption></figure><h3><strong>Model-Based Learning</strong></h3><p>Now comes the juicy section I’d like to put my energy and attention toward: <em>model-based</em> learning.</p><p>We’ve filled in our missing data and found our best informative features, so now let’s take a look at the shape of the dataset. We have 165 total data records, with 102 cases of patient survival and 63 cases of patient death. Already, we run into two major “things” –– our dataset sample size is not that large for machine learning, and we have class imbalance.</p><p>Now you could say, “Raveena, why don’t you just oversample the smaller class, or downsample the bigger class?” Indeed, I could! I could oversample the “patient-death” class with synthetic sampling, called SMOTE. (<a href="https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/">here’s a great tutorial</a> on it from sci-kit learn) However, I’m trying to incorporate <em>model-based learning</em>, which means I’ll be going along the more Bayesian route — I’ll explain that shortly below.</p><p>The basic idea is just in the name: what I’ll do is create a <em>model</em> of the top-3 features for patients who survived, and a model for those who unfortunately passed from the carcinoma. The hope is, once I have these two models, for a new test-patient, I can compare the judgements from each model and decide if the test patient will survive, die, or come to an indecision due to big uncertainty.</p><p>The Bayesian step includes one more thing before the final decision: I’ll incorporate our prior knowledge of the patients and multiply it to the model judgements. What’s the prior knowledge, you ask? Well, in the <em>training set</em> there are 73 patients who survive, and 42 who die — because of the “class-imbalance” I mentioned earlier. And it might’ve sounded like a bad thing. But in model-based learning, we can actually use it to our advantage! What I’ll do is — combine the prior differences between the surviving and dying patients, together with the model judgements — and use those two to come to a final decision.</p><h4><strong>Creating the Models</strong></h4><p>To create the actual models, I’ll be using a type of model known as a Gaussian Mixture. In the easiest way to describe it, a Gaussian Mixture model looks at a bunch of data, and tries to decompose the distribution of data as a basic sum of individual bell curves, or Gaussian normal distributions. Each simpler normal distribution has a specific weight associated with it — the weight tells the model how much importance to give that specific Gaussian. Here’s a GIF of a Gaussian mixture model fitted to data <a href="https://en.wikipedia.org/wiki/Mixture_model#/media/File:Movie.gif">linked here</a>, and one screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZtppVXlL-O9PgrjhlbAtcA.jpeg" /><figcaption>Image Credit: Wikipedia. Screenshot of a Gaussian Mixture Model and its components. You can see the individual red, yellow, purple, green and teal normal distributions, and each one is added with a specific weight to give the overall thin darker blue curve</figcaption></figure><p>Gaussian Mixture models also are less prone to overfitting the training data than kernel density estimators, which usually try to maximize the fit of a distribution to training data, making the risk of overfit.</p><pre># Gaussian Mixture model for patients who survived carcinoma<br>life_model_2feature <strong>=</strong> GaussianMixture(n_components<strong>=</strong>6, random_state<strong>=</strong>0)<strong>.</strong>fit(X_train_2life)</pre><pre># Gaussian Mixture model for patients who died from carcinoma<br>death_model_2feature <strong>=</strong> GaussianMixture(n_components<strong>=</strong>4, random_state<strong>=</strong>0)<strong>.</strong>fit(X_train_2death)</pre><p>We’ll look at the models of survived and died patients based on the top 2 features; the top-3 features is detailed in my code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*zxAMuYyrE5vGcpgl0EzvyQ.jpeg" /><figcaption>Image credit: author. This is the internal-model the computer has for survived patients. It’s done this by generating 100 synthetic samples of survived patients based on its model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*2nznABN47YWk7TwBM-gFug.jpeg" /><figcaption>Image credit: author. This is the internal-model the computer has for died patients. It’s done this by generating 100 synthetic samples of died patients based on its model</figcaption></figure><h4>Model Predictions</h4><p>Finally, it’s time to see how the model will predict patient survival on test patient data. I want to discuss one important thing before jumping right in though.</p><p><strong>We Can’t Risk False-Positives Here</strong></p><p>If we are predicting whether a patient will survive this specific carcinoma based on their Alpha-Fetoprotein and Hemoglobin content, it would be extremely dangerous and unethical to accidentally predict a patient would survive carcinoma, when they might have a high chance of dying. This is known in statistics as “false positives”, where in this situation, positive = survival and negative = death. If we’re going to make a final decision about test patients, we need to put more weight if we make a false-positive choice — we need to “punish” that decision so it happens less, and we don’t accidentally lie to patients by telling them they’ll survive the carcinoma, when their chance of dying without proper treatment is much higher.</p><p>Say we want to make false-positive decisions 5 times the weight of false-negative decisions. If we want to punish false-positive decisions 5 times as much, then we can only accept a decision to predict the patient as “survived” if the final odds-ratio is above a certain amount. More specifically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*O7k-YM10YXFii-tSw9cbjw.jpeg" /><figcaption>Image credit: author. This is basically Bayes Theorem, and to punish false-positive decision 5 times more more heavily, we make the decision stakes higher.</figcaption></figure><p><strong><em>(Just as a important reminder, in this project, Y=1 means the patient will survive, and Y=0 means the patient will not survive the carcinoma.)</em></strong></p><p>(The details to this are in the code, so feel free to <a href="https://github.com/galaxyenby1997/carcinoma-modeling">check it out yourself here</a>). But here’s a sample of code if you’re interested:</p><pre>threshold = 5<strong> </strong></pre><pre><strong>for</strong> odds <strong>in</strong> log_posterior_odds:</pre><pre>    <strong>if</strong> odds <strong>&gt;</strong> np<strong>.</strong>log(threshold):<br>        decisions<strong>.</strong>append(1)<br>    <strong>else</strong>: <br>        decisions<strong>.</strong>append(0)</pre><p><strong>So…What does the Model Predict?</strong></p><p>Implementing this decision rule, we get the following confusion matrix results. (A confusion matrix is just a fancy term for a 2x2 table which shows how the model performs on test data, with false positives, false negatives as well — <em>not just accuracy</em>. Accuracy can sometimes be misleading!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*kcHCQGuez0uouT9J4ljAtg.jpeg" /></figure><p>The model didn’t do too bad! –– around 36 correct out of 50, which gives 72% accuracy. The dark purple box in the top-right is the false-positives — the model has predicted for 5 test patients that they would be healthy, but actually would not survive the cancer.</p><p>But I want to draw your attention to something more important than accuracy for this model-based learning framework — because remember, accuracy isn’t everything. Remember how I said we would punish the model for making more false-positive mistakes (the model lies to the patient that they will survive the carcinoma) by setting the ratio threshold at 5?</p><p>Well, watch what happens to the number in the top-right purple box when I increase the threshold to 10, so we put 10 times more weight:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*8tBm7ZDd4vPP0Rh11a7M-g.jpeg" /></figure><p>By increasing the threshold to 10, the model made one less false-positive mistake, at the cost of making a false-negative mistake.</p><p>But watch what happens if the threshold is very high, say at 20. So we put 20 times more weight on false-positive classification mistakes than false-negatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*RQFMnMnpXXrI_2_NtQoyLA.jpeg" /></figure><p>Now the model has really started to minimize false-positives, which is great — but it comes at the huge cost of increasing false-negatives. This means that too many patients who might not need checkups, have been sent by the model to their medical providers for checkups. And unless their insurance can cover regular checkups, this would not be good money-wise on the patient side. So as you can see, setting the threshold is a balancing game for the model.</p><h3>Final Words</h3><p>I hope this article has given you a sense into the flexibility of model-based learning, and how it can be applied to real-world data in the health domain,</p><p>One of the major advantages of model-based learning, in my opinion is that this framework can be applied to many types of tabular data — not just health records. This type of machine learning can even be applied to vision and speech-recognition domains, as long as the right types of features are extracted from the data, and effective probabilistic generative models are used.</p><p>Another advantage I argue, is that the final analysis we did on false-positives, false negatives, and compensating for either mistake <em>based on the domain</em> <em>of our data</em>, is pretty straightforward using model-based learning. All we have to do is decide how much weight we put toward punishing false-positives or false-negatives, and set that as a threshold as we did. It just takes a few lines of code in Python, but this analysis is extremely important in for example, the healthcare domain where making a false-positive mistake might be more costly or deadly than a false-negative, <em>regardless of accuracy</em>.</p><p>Thank you so much for reading, and I’ll see you in my next article!</p><p>You can check out <a href="https://www.linkedin.com/in/sidarth-jayadev-32b321391/">my LinkedIn here</a>, if you’d like to learn more about me, what I’m interested in and what I’d like to pursue in machine learning and data science!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=581bd58a1a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/model-based-decision-making-for-health-data-581bd58a1a3">Model-Based Learning on Cancer Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Could a “Green AI” Look Like?]]></title>
            <link>https://medium.com/data-science/what-would-a-green-ai-look-like-28d91aaff3be?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/28d91aaff3be</guid>
            <category><![CDATA[environmentalism]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[probabilistic-programming]]></category>
            <category><![CDATA[data-for-change]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Fri, 05 Nov 2021 16:35:26 GMT</pubDate>
            <atom:updated>2021-11-24T15:57:29.140Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/data-for-change">Data for Change</a></h4><h4><em>What could an environmentally-friendly AI look like in the future?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iYhdlwM4D00olMvC" /><figcaption>Photo by <a href="https://unsplash.com/@chelseadeeyo?utm_source=medium&amp;utm_medium=referral">Chelsea</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Disclaimer: This is an opinion piece, and the opinions expressed here only belong to the author.</em></p><p>You’re probably asking, “Raveena, just what exactly do you mean by a <em>Green </em>AI?” Well, I guess I did reveal my intention in the italic summary above, but that doesn’t exactly give us a solution to what environmentally-friendly AI could look like, since none of us can accurately predict the future 10 or 20 years––or even 5 years from now. Heck, just 9 years ago, the first massive deep neural network was created, able to bang out extreme record-breaking results on image classification––think of the ImageNet competition. Neural networks could now accurately classify <em>real</em>, digital images of tens of species of dogs, animals, objects, and the like. I’ll be honest with you right now, I had NO clue that “maltese dog” and “Tibetan terrier” were <em>actual dog breeds</em> until I peeked at the labels in the ImageNet dataset!</p><p>Over the last few years, we’ve built even better deep learn systems that can not only classify images but work with videos––I never would’ve expected that––and in my opinion, the most mind-blowing results–– creating text-to-image captions, and language models such as GPT-2, BERT and GPT-3 which have produced truly stunning results on producing human-like readable paragraphs and sentences.</p><p><em>But is this the most efficient type of AI we can create––is this really progress toward “greener AI”?</em></p><p>I argue that in fact, it’s probably <em>not</em> the most efficient type of AI we can create, and I also believe that creating more and more deeper network models with more complicated architectures, isn’t really paving a road toward “green AI.” For one, although these models may be beasts when it comes to language generation, caption generation and other tasks, they are also beasts of <em>energy usage</em>.</p><p><a href="https://dl.acm.org/doi/pdf/10.1145/3442188.3445922">In a recent paper by Dr. Timnit Gebru and her colleagues</a> — Dr. Gebru, a prominent Black AI researcher at Google Brain, who was <em>fired</em> for reporting this — cited <a href="https://arxiv.org/pdf/1906.02243.pdf">this computational linguistics paper by Dr. Emma Sturbell</a>, stating that training a “<em>single</em><strong><em> </em></strong><em>large</em><strong> </strong>language transformer model<strong> </strong>from scratch” was estimated to use <strong>around 300 <em>tons </em>of CO2</strong>, while humans individually are responsible around 5 tons of CO2 a year. Just to clarify, a “transformer model” is just a fancy word for “big” deep learning language model.</p><p>Neural architecture search is another newer technique in deep learning — but just think of it as a computer “searching” for possible ways to arrange the nodes and connections in a neural network. It might sound sexy, but it too is an energy beast. In the same paper, Dr. Sturbell and her colleagues calculated that training a single transformer model using a technique called “neural architecture search” used electricity and energy consumption equivalent to <strong>626 thousand tons of CO2 emissions — </strong>or the same as CO2 emissions as driving five Teslas at the same time until their<em> lifetime capacity was burned up.</em></p><p>Training a <em>single</em> BERT model<strong> </strong>from scratch would require the same energy and carbon footprint as a<strong> commercial, passenger trans-Atlantic flight</strong>, according to Sturbell.<strong> </strong>More ominously, Dr. Gebru and her colleagues wrote that “massive Internet-based training data for large language models has been shown to have <em>problematic characteristics </em>resulting in “<strong>models that encode stereotypical and derogatory associations along gender, race, ethnicity, and disability status”. </strong>Finally,<strong> </strong>Sturbell notes the economic costs of training these large NLP models, saying “many [NLP models] now require multiple instances of specialized hardware such as GPUs or TPUs, therefore <strong>limiting access to these highly accurate models on the basis of finances</strong>. Basically, if you don’t have enough money to rent GPU usage or buy GPU chips to perform deep learning––you can’t really do innovation or computer with large NLP models unless they’ve been specifically pre-trained.</p><p>Now, you could say, well, these models are only trained once, and then deployed to the cloud and can be used by others. However, deploying large deep-learning models to the cloud costs a lot of energy, as <a href="https://www.forbes.com/sites/moorinsights/2019/05/09/google-cloud-doubles-down-on-nvidia-gpus-for-inference/?sh=6a20cc636792">cited by Forbes in this article</a> about NVIDIA––around 80% of the energy used for a model goes to <em>inference alone</em>. Google, Facebook, and others are big corporations, with more money and resources than you or I will ever have in our lifetime. Should we really trust large corporations to democratize AI for all of us, while they take oceans of data from us — with almost minimal consent — to train their newer, sexier, larger models?</p><p>The final point I’ll say about deep learning‘s limitations <em>by itself </em>is this: traditional deep-learning neural network models don’t have a good way of dealing with uncertainty. Here, let me give you an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PAb_UQ1s3_EuomP0" /><figcaption>Photo by <a href="https://unsplash.com/@bubo?utm_source=medium&amp;utm_medium=referral">Lubo Minar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here, our human brains can tell with a large confidence that the photo above is of a metropolitan, perhaps downtown section of a big city — even though the photo is blurred. But traditional convolutional neural networks don’t have an in-built mechanism for displaying their own “network uncertainty” when it comes to image recognition.</p><h3>So, What Can We Do?</h3><p>So what’s to be done? I’m not implying in any way that deep learning is bad — deep learning is truly an engineering marvel in itself — but <em>by itself</em>, deep learning hasn’t really come close to the generalization flexibility of the human brain over a large variety of tasks. And proclaiming that deep learning is the <em>singular solution to AI and intelligence</em> —personally, that’s a bit presumptuous to say.</p><h4>Towards AI Flexibility</h4><p>What we need, in my honest, humble opinion, is something cleverer than pure function approximation: we need to understand how inference and model-building works in the brain. Admittedly, this is like passing the “hot potato” down the line — having to answer probably a harder question than “How to democratize AI?” and “How to make a Greener AI?”, but let’s go with the flow and entertain this question.</p><p>If you’ve ever taken the SAT, you probably remember those dreaded questions called “sentence inference” questions — where a incomplete sentences is given, with a blank — and you have to fill in the blank (or many blanks) with the “best fit” word. So maybe an SAT exam example would be like this:</p><blockquote>SAT Example Question:</blockquote><blockquote>Since Kelly didn’t expect anything for Valentine’s Day, she was _________ by the bouquet of roses on her desk.</blockquote><blockquote>From the answer choices: 1) intrigued, 2) astonished, 3) love-struck, 4) disappointed, 5) irritated –– which word would you choose? What would your answer be here?</blockquote><p>The answer to this question is (2), and I’m sure that might have been “obvious” to all of you, but would it really be that obvious to the AI systems we currently have for natural-language processing? Many solutions could grammatically work, and there might be way to solve this using the statistics of frequently occurring words — but solving this efficiently — the answer we’re looking for best fits the context and meaning of the sentence, and also fits our intuition for how Kelly would feel.</p><p>Now, this example might not seem to involve probabilities on the surface of it — after all these are SAT words! But think about this: if you didn’t know much about the SAT and hadn’t studied for it, and just saw this question, you’d be making an educated guess<em> </em>right? You know, prior to seeing the question that Valentines is associated with roses, and we also have a prior intuitive model<strong> </strong>for how Kelly might react, given the information in the sentence and the fact that we have no other information about Kelly’s intentions, beliefs or emotions. This type of high-level, flexible, model-based <em>abstract reasoning</em> is something human brains are especially good at — and today’s AI systems are not really near this level, especially when it comes to AI trying to model the behaviors of other humans.</p><blockquote><strong>This type of high-level, flexible, model-based abstract reasoning is something human brains are especially good at — and today’s AI systems are not really near this level, especially when it comes to AI trying to model the behaviors of other humans. Green &amp; responsible AI needs this.</strong></blockquote><p>But in the last 10–15 years, cognitive scientists, cognitive psychologists and AI researchers together, have been making headway into creating machine systems that can do this type of reasoning: recently, it’s been called <em>probabilistic programming</em>. I won’t go into too much detail, so if you want to learn the basics of it before returning to this article, check out the <a href="https://towardsdatascience.com/september-edition-probabilistic-programming-11a6699e2fac">TDS Editors’ September Edition article</a> on it.</p><p>In my humble opinion, probabilistic programming offers a pathway to environmentally-friendly AI, because it creates a possibility to express a large amount of reasoning without the need for brute-forcing inference with thousands or millions of data points. When you see a picture of a car, or a motorcycle, you don’t need to know the hundreds or thousands of exact pixel locations and values in the photo: instead, you have some idea of the <em>primitives</em> of the object. The car wheels, chassis, windows, general edge shape, mirrors, an engine hood — these are all the “primitives” or <em>building blocks</em> of the object. With these building blocks, a human could draw almost an <em>infinite </em>variations of a car. In fact, researcher and computer graphics expert Daniel Ritchie has used probabilistic programming to generate realistic 3D virtual spaceships and other objects from just a few examples and <em>primitives — </em>the building-blocks — of the objects. And he details this process in an <a href="https://www.youtube.com/watch?v=QprVu_pLT4w">amazing video from the 2020 Probabilistic Programing Conference</a>, where he uses a graphics program to render many variations of spaceships using its primitives — for example, the body of the ship, the wings, the thrusters.</p><blockquote>When you see a picture of a car, or a motorcycle, you don’t need to know the hundreds or thousands of exact pixel locations and values in the photo: instead, you have some idea of the primitives of the object. With these simple building blocks, a human could draw almost an infinite variations of a car.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-HXq7ObdeRB3XhAuhjslg.jpeg" /><figcaption>A Probabilistic Program could identify abstractly the sub-parts that make up a dog, and use generative models and the idea of — “the generated whole is the sum of generated parts” — to create new examples of dogs. (Image credit: <a href="https://en.wikipedia.org/wiki/File:Whole_Dog_vs._Dog_as_Sum_of_its_parts.jpg">https://en.wikipedia.org/wiki/File:Whole_Dog_vs._Dog_as_Sum_of_its_parts.jpg</a>)</figcaption></figure><h4><strong>Coming Back Full Circle</strong></h4><p>Speaking of Prof. Ritchie’s computer graphics work, I do want to come back full circle, to the beginning of this piece. In one of Ritchie’s work in 2016 (and in the video linked above), he notes that performing <em>inference</em> in probabilistic-programming — that is, inferring the hidden generative programs, especially in the context of graphics, given the probabilistic model, is a very complex task. The usual inference techniques include things called “random sampling” methods — but by themselves, these take quite a long time to approach an answer. In the video talk, Prof. Ritchie talks about computer-generating graphical representation of alphabets, based on a single sample image.</p><p>Using random sampling, the inference took about 10 minutes — but Ritchie was able to combine the probabilistic graphics program with <em>another tool</em> to generate the image <em>10 times faster</em>. Now — what was this tool, you may ask? Coming back full circle, it was — lo’ and behold, a neural network. Turns out, neural networks are particularly good when it comes to the inference algorithms part of probabilistic programming — the network was able to “guide” the program toward finding faster values that optimized the graphics program.</p><h3>Conclusion</h3><p>I hope I’ve been able to show you, the reader, some of my thoughts around what a environmentally-friendly type of AI could look like in the near future. Of course, if we could predict the future accurately, time would be kinda meaningless, but if I had to take a guess — creating a greener AI would involve deeply understanding how the brain is able to perform intelligence in many ways that current AI methods are lacking. Green AI will have to involve hard-coding the types of abstract reasoning human brains can do, so we don’t have to store huge amounts of data in data centers and spend hundreds of thousands of dollars to train-from-scratch, new language models or other AI language models to imitate humans. Green AI also needs to have <em>uncertainty</em> built-in, to deal with the fact that the real world is messy, and not at all clean like the logical statements of mathematics. And crucially, Green AI will need input not just from computer scientists and mathematicians, but from cognitive science, cognitive psychology, neuroscience, cultural psychology, and many other diverse areas.</p><blockquote><strong>Green AI will need input not just from computer scientists and mathematicians, but from cognitive science, cognitive psychology, neuroscience, cultural psychology, and many other diverse areas.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28d91aaff3be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-would-a-green-ai-look-like-28d91aaff3be">What Could a “Green AI” Look Like?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s in a “Random Forest”? Predicting Diabetes]]></title>
            <link>https://medium.com/data-science/whats-in-a-random-forest-predicting-diabetes-18f3707b6343?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/18f3707b6343</guid>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest-classifiers]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Sat, 02 Oct 2021 01:43:56 GMT</pubDate>
            <atom:updated>2021-10-06T23:14:38.451Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a>, Machine Learning, Intuited</h4><h4>Understanding a Random Forest intuitively and predicting diabetes in real hospital patient data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zerQYw-7FPRHJFAD" /><figcaption>Photo by <a href="https://unsplash.com/@priscilladupreez?utm_source=medium&amp;utm_medium=referral">Priscilla Du Preez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>(<em>Disclaimer:</em> This article is mainly for people who want a friendly, intuitive understanding of what’s going on in random forests and decision trees — so I won’t be going into huge mathematical detail. I’ll post some links further below in the article to videos and material if you want to go more into the statistical details. If you’re also interested in a Python implementation, read further!)</p><p>If you’ve heard of “random forests” as a hot, sexy machine learning algorithm and you want to implement it, great! But if you’re not sure exactly what happens in a random forest, or how random forests make their classification decisions, then read on :)</p><p>We’ll find that we can break down random forests into smaller, more digestible pieces. As a forest is made of trees, so a random forest is made of a bunch of randomly sampled sub-components called <em>decision trees</em>. So first let’s try to understand what a decision tree is, and how it comes to its prediction. For now, we’ll just look at classification decision trees.</p><p>Decision trees have, as the name says, a tree-like structure that is made up of things called “True/False tree-branches” that split (just like branches in real-life trees split!). It’s like looking at an upside down tree — imagine the trunk of the tree at the top. In fact, a decision tree is very interpretable, as humans make decision trees too about certain choices — more commonly called flowcharts!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*saZWqFLll7K0nvnDz9DPAw.jpeg" /><figcaption>Image credit: Wikipedia (<a href="https://en.wikipedia.org/wiki/Decision_tree_learning#/media/File:Cart_tree_kyphosis.png">https://en.wikipedia.org/wiki/Decision_tree_learning</a>). Here we see a simple flowchart (decision tree) for how a computer estimates the probability that someone survived the Titanic sinking</figcaption></figure><p>In the decision tree above, the machine (or perhaps a diagnostician) makes a series of judgements at every level of the tree, going down further until the probabilities are reached. For example at the top root of the tree: the machine asks: is it True or False the passenger’s gender is male? If it’s true, go down the left branch; False, go down the right branch. This is an example of a True/False (or Yes/No question) in a decision tree.</p><p>In this way, it’s very easy to visualize a decision tree. But how exactly do decision trees, you know, <em>make</em> their decision?</p><p><strong>Choosing What “Features” to Split On</strong></p><p>In the Titanic data example pictured above, we see that the decision tree decided to make “splits” in the tree for the passenger in question (the True/False question split I mentioned earlier) based on certain features, like the <em>gender</em> of the person, their <em>age</em>, and the <em># of siblings</em> of the passenger. How did the decision tree figure out these features? Because we didn’t explicitly ask the tree to pick those attributes.</p><p>Imagine yourself as the decision tree, and you’re faced with predicting if a passenger from the Titanic died or survived based on the data. At first, when you haven’t asked Yes/No questions about the attributes, you don’t know how to sort the “dead” passengers and “alive” passengers. But intuitively, if you pick the <em>right</em> features to ask Yes/No questions on (like the above graph shows), far enough down in your flowchart decisions, you’d probably want one branch to have all the “dead’ passengers and a separate branch to have the “alive” passengers, right? At this point, you’ve sorted and filtered out your passengers, and by picking the right attributes, you can literally <em>read</em> the flowchart, visually follow each branch downward and see if the passenger is alive or dead — what an awesome way to make a prediction!</p><p><strong>What is “Information Gain” in a decision tree?</strong></p><p>Picking features so that certain “branches” of the decision tree become “full” of just one type of data — for example, one branch just has all the “dead” passengers and another branch has just the “alive” ones — that notion of “fullness” is directly related to a concept in probability called <em>information content</em>.</p><p>To give some motivation and intuition for “information content”, imagine this situation: you lived in an area plagued by lack of rainfall, and because of this, dry grass and plant matter often caught on fire more easily from the Sun’s exposed heat — you started becoming used to seeing fires around the local area. However, one night, you suddenly hear a <em>crack!</em> in the sky, following by a huge explosion of fire, and, visibly shocked you realize a lightning strike caused the fire this time. You had never seen lightning striking in the local area! It was more <em>surprising</em> to see a lightning cause the fire this time. That word, “surprise”, is the <em>information content</em> of the event — in this case, it was lightning causing the fire (instead of the Sun’s heat+dryness causing a fire to spread). The surprise was high because you didn’t expect the lightning to be a cause — in the same way, the information-content of an event is <em>higher </em>if the event was <em>less likely</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g4YGRmXARNtcExPi" /><figcaption>Photo by <a href="https://unsplash.com/@davidmoum?utm_source=medium&amp;utm_medium=referral">David Moum</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nOMSEZsQDeZMiao7" /><figcaption>Photo by <a href="https://unsplash.com/@werksk?utm_source=medium&amp;utm_medium=referral">Dominik Kiss</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We can call “cause of the fires in a local drought area” a <em>variable</em>, something we don’t know. There are many values of this variable, since there can be many causes. Some of those values (or call them <em>events</em>), like “a small spark lit up thousands of dry parched grass in the area and a fire spread” are more likely than “a sudden lightning strike caused the fire spread.” To get a better general sense of this variable, we can average out values that have higher chance, and lower chance, and derive a number called the <em>entropy</em> of the variable. This number, entropy, is crucial to understanding how decision trees make their Yes/No question splits.</p><p><strong>How the Decision Tree Makes Its Classification</strong></p><p>Remember how I said before, your intuitive goal in being a decision tree classifier is to pick the “right” features so that as you go down the tree in your Yes/No (or True/False) decisions, at the end you should have a branch or two that’s just purely made up of the “dead passenger” class and another one or two branches with the “alive class”? In a sense, by asking the right Yes/No questions — choosing the right feature attributes — we’re traveling down the tree of our choices and making the last branch <em>pure</em>. In a decision tree we want to choose features to increase the class “purity” as we travel down the branches.</p><p>Now, if the final branches contain all, or most of the “alive passenger” class, notice something: the <em>proportion</em> of “alive” to “dead” passengers in that branch is very uneven; there’s way more alive ones. In the lightning-strike analogy, the “alive passenger” class has “high purity” — which is also called “<em>low</em> entropy” in that “final branch” — why? Because the “alive” class is much more likely in that branch; there’s just more “alive-class” data points there! In a symmetrical way, in the final branch for “dead-class” data points, ideally the entropy for the “dead” category is also low, because that branch is purely made up of “dead passenger” data. So our goal in the decision tree is to find the “right” features to split so we can <em>lower</em> the entropy as much as possible using True/False questions. This “lowering of entropy” down the decision tree is also called “increasing <em>information gain</em>”. It’s called this because reducing the entropy in the tree helps you gain information about finding the right features to split on, and therefore creating a good classifier.</p><p><strong>Many Decision Trees Together= Random Forest!</strong></p><p>Now that we’ve gone through an intuitive feel for how a decision tree works, we do need to mention a few things. Decision trees are very interpretable (because you can literally read them left-to-right and top-to-bottom) but they are prone to <em>overfitting</em>. Overfitting happens when machine learning algorithms fit their model too closely to the data, so they can’t generalize well on test sets.</p><p>How can we fix overfitting? Well, instead of looking at just one decision tree, we look at a <em>collection</em> of them, say 10 or 20. Depending on the classification from different decision trees, we let the trees vote. Say we are classifying diabetes in hospital patient records. If we train 20 decision trees on random subsets of the data, and for a new, <em>un-seen </em>patient record, 15 of trees say “Yes, this patient has diabetes!” and only 5 trees say “No!”, the majority vote (like in a democracy!) defaults to predicting diabetes for the patient. This “collection” of decision trees making a democratic vote on a classification — is the <em>random forest</em>.</p><p>One important note I want to make before going onto Python implementation: the 20 decision trees in the example above are trained by <em>sampling subsets</em> of the data; each decision tree is built from a different random sample of the data — this is called <em>bootstrapping </em>(it comes from a term in statistics called bootstrap sampling — when you need to get information about your population but you only have a small sample to work with; say, a small community survey.) When we do the “democratic vote” among the 20 decision trees in the random forest, that’s called <em>aggregating</em> — and so combining these two methods is called <em>bootstrap aggregating</em>, or “bagging.” If you’d like to understand more details about this “bagging” process in random forests, check out <a href="https://towardsdatascience.com/bagging-decision-trees-clearly-explained-57d4d19ed2d3">this post on Medium</a>!</p><p>If you’d like to see a video going more into the mathematics of decision trees than I could in my post — so you can get a better solid understanding of information gain, entropy, and the concepts we discussed above — check out <a href="https://www.youtube.com/watch?v=7VeUPuFGJHk">this awesome video</a> by the YouTube channel “StatQuest” :)</p><p><strong>Python Implementation</strong></p><p>Now that we’ve gone through some conceptual context behind what a random forest and a decision tree is, and how it makes its decisions, let’s actually implement the algorithm in Python!</p><p>For this implementation, I’ll be using <em>real-life recent data</em> from patients at the Sylhet Diabetes Hospital in Sylhet, Bangladesh. The data was collected and published just last year in June 2020, in this research paper by Dr. MM Faniqul Islam and others (cited below), and is freely available on the UC Irvine Machine Learning Repository <a href="https://archive.ics.uci.edu/ml/datasets/Early+stage+diabetes+risk+prediction+dataset.">at this link</a>.</p><p>First, you’ll need to import the CSV file once it’s downloaded from the Repository. The dataset, once formatted in the Pandas Data-Frame module, should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGwMDp8eiQTIvXP51Kfwtw.jpeg" /><figcaption>Image credit: Raveena Jayadev, author</figcaption></figure><p>Once we’ve downloaded the data, it’s good standard practice to convert the “Yes” and “No” answers into 1’s and 0’s, respectively in a numerical format. We can do that by coding the following:</p><pre>binary_dict = {&#39;Yes&#39;:1, &#39;No&#39;:0}<br>gender_dict = {&#39;Male&#39;:0, &#39;Female&#39;:1}<br>DBdata_copy[&#39;Gender&#39;] = DBdata[&#39;Gender&#39;].map(gender_dict)<br>for name in DBdata.columns[2:-1]:<br>    #loop through each column of the dataframe<br>    x = DBdata[name]<br>    DBdata_copy[name] = x.map(binary_dict)</pre><p>Anyway, once we’ve converted the data numerically, it should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOpDESD3RocanUwvDf1hnw.jpeg" /><figcaption>Image Credit: Raveena Jayadev, author</figcaption></figure><p>All we have to do now is use the random-forest classification models from Python’s awesome Sci-kit Learn’s module. We can instantiate the classifier like this:</p><pre>from sklearn.ensemble import RandomForestClassifier</pre><pre>rf_classifier = RandomForestClassifier(n_estimators=20, criterion=&#39;entropy&#39;, n_jobs=-1)<br>rf_classifier.fit(X_train, y_train)</pre><p>Notice one of the parameters in the “RandomForestClassifier” module: <em>criterion</em>. Where I’ve put <em>criterion=’entropy’</em>, that’s the same “entropy” word I was talking about earlier when explaining how random forests pick their features to split on.</p><p>Awesome! Once we fit our random forest classifier, we can plot one of the most important visualizations for forests: <em>feature importances. </em>This can be done using the bar-plot (<em>plot.barh</em>) in Matplotlib, along with this piece of code: rf_classifier.feature_importances_</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKV21nie72libOIfsu-cHQ.jpeg" /><figcaption>Image Credit: Raveena Jayadev, Author</figcaption></figure><p>It looks like “Polydipsia”, “Polyuria”, and age were 3 of the most important factors the Random Forest used to classify diabetes. Awesome!</p><p><strong>Visualizing the Actual Decision Tree</strong></p><p>Now here comes the fun part that probably all of you have been waiting for: we’re going to visualize one of the decision trees in this random forest. To do this, first we’ll need the package “<em>graphviz</em>”. Graphviz is a visualization tool developed for decision trees, and you can install it by using pip install graphviz or, if you’re in an Anaconda environment, conda install -c graphviz.</p><p>W can choose one of the individual “estimators” in the random forest — the estimator being a decision tree, and program the visualization like this: (here, the decision tree I chose, I called it “estimator_0”.</p><pre>estimator_0 = rf_classifier.estimators_[0]</pre><pre>tree_estimator = export_graphviz(estimator_0,<br>                 feature_names=features,<br>                 class_names=target)</pre><pre># Draw tree<br>graph = graphviz.Source(tree_estimator, format=&quot;png&quot;)</pre><p>And the decision tree looks something like this! You should be able to click on the image and zoom into it with an additional tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6sBJCeYJIT42fsbUgbnwA.png" /><figcaption>Image Credit: Raveena Jayadev, Author</figcaption></figure><p>Let’s look at the word “value” in each of the boxes: what does that mean?? Well, when you see something like value=[149, 215] all the way at the root node, it means that before the True/False feature split, 149 samples were of the “patient-has-no-diabetes” (Negative) class and 215 were of “patient-has-diabetes” (Positive) class. If you recall from the “Information Gain” section of this post, I mentioned that the goal of a decision tree is to make the “best” True/False splits so right at the very bottom of the tree we get these “pure” nodes where there’s only one type of class or another. The trees aren’t perfect so the bottom-most nodes aren’t always 100% pure, but if you look at the bottom in the picture above, for example: at the “Age ≤ 39.5, entropy=0.8” split near the bottom, the values say [0,2] in the blue box on the left and [6,0] in the orange box on the right. This means that feature split of Age≤39.5 was good enough to create pure nodes where on the left, it’s only positive-diabetes patients and on the right, it’s negative — this is fantastic! As you can see, the farther down the decision tree, the better the splits into pure nodes.</p><p>Another easy, visual way to see how the “purity” of each node in the decision tree increases — and hence, a better feature split — is that the darker the hue of the box — so dark blue, or dark orange — the purer the node. Notice how at the top of the tree, nodes are generally light-brown/light-blue/light-orange, and as you go down, the color of the node becomes darker.</p><p>In terms of accuracy, random forests do a fantastic job because they <em>average</em> over many decision trees. By themselves, decision trees overfit but as a team, they make great majority-rule classifications. In our example, the random forest was 94% accurate, with a false-negative rate of 2.2%, which is great for a detector of diabetes, since it’s more risky for a machine to make a false-negative, and lie to a patient saying they don’t have diabetes.</p><p><strong>Conclusion</strong></p><p>I hope you, the reader, have gotten a better intuitive feel for random forests, decision trees, and how they make their classification decisions. I also hope you’ve learned the basics of implementing the code in Python and interpreting each decision tree. You can check out <a href="https://github.com/galaxyenby1997/Diabetes_ML_RandomForest">my Github code here</a> for downloading the diabetes data and importing <em>graphviz</em> for the visualizations of the forest.</p><p>See you next time! :)</p><p><strong>References</strong></p><ol><li>Islam, MM Faniqul, et al. “Likelihood prediction of diabetes at early stage using data mining techniques” (2020), <em>Computer Vision and Machine Intelligence in Medical Image Analysis</em>. <em>Springer</em>, Singapore. pp. 113–125.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18f3707b6343" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/whats-in-a-random-forest-predicting-diabetes-18f3707b6343">What’s in a “Random Forest”? Predicting Diabetes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programming An Intuitive Image Classifier, Part 1]]></title>
            <link>https://medium.com/data-science/programming-an-intuitive-image-classifier-part-1-266bd657aa4?source=rss-cb567727317e------2</link>
            <guid isPermaLink="false">https://medium.com/p/266bd657aa4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sidarth Jayadev]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 15:37:18 GMT</pubDate>
            <atom:updated>2021-09-20T16:43:51.221Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a>, <em>More Intelligent than Artificial</em></h4><h3>Engineering Intuitive Computer Vision</h3><h4><em>How can we create an image classifier system that makes decisions and predictions like a human would?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BdvWjeDTJsMnaqaj" /><figcaption>Photo by <a href="https://unsplash.com/@maximalfocus?utm_source=medium&amp;utm_medium=referral">Maximalfocus</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Image classification is one of the hottest fields of machine learning, data science, and AI, and often used to benchmark certain types of AI algorithms — from logistic regression to deep neural networks.</p><p>But for now, I want to take your mind away from those hot techniques, and ask ourselves a question: if us humans saw an image of a handwritten character, or a dog or cat, how would our brains <em>intuitively</em> classify different types of images? Below is an example of digits in an image; “2”, “0”, “1” and “9”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8A_EfkvyxafZjdv7" /><figcaption>Photo by <a href="https://unsplash.com/@nordwood?utm_source=medium&amp;utm_medium=referral">NordWood Themes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the example above of digits (or numbers/numerals), how would our brains differentiate between, say, the 1 and 9 at the bottom? Well, intuitively our brains have a sort of “mental model” of what 1s look like, and a mental model of what 9s look like. For a new image of a digit, our brains can compare the digit based on each mental model and decide if it’s a 1, a 9, or an entirely different handwritten character. In fact, our mental model can do more than just classify: it can <em>generate </em>new instances of similar data. Show young children digits for the first time so their brains can form a mental model, and I’m pretty sure they can learn to draw <em>new</em> instances of, say a 1 or a 9, or the other digits in the photo above.</p><p>Now <em>this</em> type of decision-making, classification, is quite different from what you see in popular AI vision systems deployed around the world in cameras, hospitals, etc. — those rely on something called convolutional deep learning — <a href="https://poloclub.github.io/cnn-explainer/">check this visual demo out.</a> While the convolutional-only layers of a convolutional neural network (CNN) might resemble specific parts of the vision system in our eyes <a href="https://www.nature.com/articles/srep32672">according to this author</a>, the actual decision making of a deep CNN network mainly comes from the thousands of tuned weights in the feedforward “final fully connected layer” of the network.</p><p>BTW, if you haven’t heard of neural networks or not familiar with how neural networks are used to classify images, I definitely <a href="https://www.youtube.com/watch?v=aircAruvnKk&amp;t=985s">recommend this video by Grant Sanderson</a> on the 3Blue1Brown (3b1b) Youtube channel; his exposition and explanations are fantastic.</p><p>But the ML system we’ll be engineering to decide and classify digits will not be incorporating thousands of weights and parameters as is the case in artificial neural networks.</p><p>Before we go on, we need to clarify what a “mental model” of digits in the computer would look like.</p><p><strong>What is a “Mental Model” in the Brain vs. Computer?</strong></p><p>The short answer is that the actual mental models in the brain — well, we’re still figuring it out. There’s a lot we don’t know yet about the brain, and neuroscientists, cognitive scientists, and others have a lot to chime in.</p><p>However, there are theories growing about what mental models in the brain could resemble <em>computationally</em>. <a href="https://www.frontiersin.org/articles/10.3389/fncom.2018.00090/full">This article</a> by computational neuroscientist Karl Friston talks more about the computational theory of how brains “call on an internal, mental generative model of the world” to <em>infer</em> and <em>predict</em> the causes behind the sensations signals it receives from the rest of the body, including the visual system. In this theory, the brain basically has to infer the causes of sensations based on a mental model it has about which probable causes lead to probable bodily perceptions. Friston also mentions that a wide range of <em>real</em> biological physiological and behavioral phenomena has been predicted by computational models like this. I’m leaving out a bunch of of details, so if you’re interested, read the article :)</p><p>In our machine learning case, we’re thankfully not having to deal with the whole host of body sensations — just 28x28 black and white images on a screen. But it’s still a challenging problem! — otherwise computer vision would have been “solved” already. So now the issue remains: how do we create, in a sense, a “mental generative model” the computer can hold as an object in its memory?</p><p><strong>Generative Models as “Distributions”</strong></p><p>Earlier I said how a “mental model”, or a generative model, allows you to basically create <em>new</em> instances of similar data. If I trained a generative model on, say, the digit 2, then the computer would be able to draw new instances of a 2. Having a model to create these instances is like a rough blueprint: it allows you to fill in the details and create different instances of houses, buildings, etc. This “rough blueprint” is called a <em>distribution</em>, because, intuitively, you basically are generating a collection of new data — and distribution is just another word for collection. The technical term for this is called a “<em>probability distribution</em>” (don’t worry, I won’t be revisiting the horrors of stats class!) because in your generated collection, some samples are more likely than others. For example if the computer were generating new instances of a “2”, then a sample of 2 with the “curly hook” missing at the top would be calculated as “less likely to resemble” a 2, than if it had a 2. Being less likely just means the model will generate that type of 2 <em>less</em> than the usual way a 2 looks.</p><p>Another way of looking at generative models in the language of machine learning is that they create models of “clusters” — they help cluster the data into specific regions. This would be called “unsupervised learning” — inferring structure within the data <em>without</em> any training labels to teach the machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4X1X4fv3PUdF1iX_" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_source=medium&amp;utm_medium=referral">NASA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>. See how there are certain “clusters” of city lights that clump together in the photo above? A generative model would be able to provide the user a “blueprint model” of how light usage is distributed around cities, vs rural areas.</figcaption></figure><p>Now that we know generative models are distributions, what we need to do first is see how to train a mental model over each of the 28x28 images of digit types (0,1,2,..9). In Python’s <em>sci-kit learn</em> library, there’s a function called <em>density estimation model </em>that will help us.</p><p><strong>The Final Step in Classification: Inference</strong></p><p>Now our computer calls on its own internal, generative “mental model” for each type of digit, and generate new types of images — and that’s all cool for sure! — but we still need to finish our first goal: classify and predict new images of digits it hasn’t seen before.</p><p>The key to classifying and predicting in our intuitive classifier is called <em>inference</em>. Our brains do it all the time, actually! Imagine the following question:</p><p>“You’re only gonna be eating greens for dinner today!” What does greens refer to?</p><p>Well, green is a color, and greens used in that plural way, it’s inferred that it refers to green objects — but <em>what </em>objects? our brains <em>infer </em>that the “objects” mean green vegetables like broccoli, spinach, kale, etc. because what other green things can be cooked and eaten? Our brains easily do inferences like this all the time, but getting a natural language processing (NLP) computer system to do inferences like this is very hard without billions of data points, parameters , and high energy usage— something human brains don’t really need. Luckily we’re not dealing with NLP inference for now, but don’t get me wrong — visual human-like inference is also a longstanding challenge in AI to overcome.</p><p>Anyway, back to vision. We want our computer to <em>infer</em> what the new digit will be, based on its “mental models” (distributions) for what digits look like. How would such an inference look like? Well, let’s say we were only dealing with two types of digits; a 2 and a 7. Suppose the computer has created distributions for what it thinks 28x28 images of 2 and 7 look like. When a new un-seen image comes in, how does it infer? The computer asks itself: under which mental model is the image <em>more likely</em>: my model of a 2, or a 7? If it’s more likely under a 2, then I’ll decide the image is a 2; otherwise I decide 7. In this way, the actual classification has turned into an if-else condition statement, a statement very pervasive in computer science languages from Python, to Java, to C. In an interesting spin, the decision is quite programmatic!</p><p>If you want to learn more about probability distributions or get familiarity, I recommend <a href="https://www.youtube.com/watch?v=ZA4JkHKZM50">this video by Grant Sanderson</a> from the “3Blue1Brown” Youtube channel, along with KhanAcademy videos on probability and statistics!</p><p>So…let’s begin the implementation! Feel free to take a pause before we dive right in!</p><p><strong>Data Importing</strong></p><p>The first order of business is to import the handwritten digit data from somewhere. For that, TensorFlow and Keras (the high-level language running on top of Tensorflow) have a beautifully organized collection of datasets, and Keras happens to have handwritten digits. The MNIST dataset <a href="https://keras.io/api/datasets/mnist/">can be found here</a> (and its license is: CC BY-SA 3.0). To import the digit dataset, we just use the code:</p><pre>from keras.datasets import mnist<br>(x_train, y_train), (x_test, y_test) = mnist.load_data()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*RY5MWliPM198ixDCQaj0RQ.jpeg" /><figcaption>Here’s a sample of the Keras MNIST training image data</figcaption></figure><p>The MNIST dataset from Keras has 6,000 training samples per digit-class and we have 10 classes (so 60,000 training digits total), but we won’t be using that much data. Why? Well, for one, to show in principle we can create these generative models without as much data. But also, in the real world, 60,000 training data points might not exist — there might only be tens or a few hundred training data per class. So what we’re going to do is just select some random group of, say 6,000 digits from the training group. That gives us about 600 training digit samples per digit-class. That’s already a <em>decrease of 90%</em> of the original data! So if we can train generative “mental” models of each digit-class based on this smaller training set, we would be more data efficient.</p><p>We can randomly sample and create a new smaller training set with the following code:</p><pre>np.random.seed(10)<br>random_indices = np.random.choice(60000, size=6000, replace=False)<br>random_x_train = x_train[random_indices, :]<br>random_y_train = y_train[random_indices]</pre><p><strong>Feature Extraction</strong></p><p>In order to create the models, we first have to do what’s called “feature engineering” (or “feature selection” if you’re working with data where the features are columns of a table). Why do we have to do this? Well, we have to realize something: these images live in a <em>very, very </em>high-dimensional space<em>. </em>Each image has 784 pixels, so as a <em>vector, </em>each image lives in the same 784-dimension space. We can’t even plot a graph in 4D space for that matter, so just give up on graphing in 784D space!</p><p>Working with data in high dimensions can be pretty difficult, and high-dimensional space does some weird, weird things — for example, spheres start to become “spiky” in 4D and higher: <a href="https://www.youtube.com/watch?v=mceaM2_zQd8&amp;t=224s">check out this video by Youtube channel Numberphile</a> if you want to get a taste. You might have also heard of this phenomenon as the “curse of dimensionality” in machine learning.</p><p>If you have tabular data with features as columns, there are statistical ways to pre-select a smaller set of features based on statistical stuff like correlation and spread. If you <em>don’t</em> have tabular data, in general there are a few ways to do feature <em>extraction. </em>Luckily for us, there’s an amazing way to extract features — also called “dimension reduction” in the ML lingo — where we can save almost all the information about the data and go do all the way to 2D! It’s called UMAP — y<a href="https://umap-learn.readthedocs.io/en/latest/">ou can read the Python user manual here</a> — and it was created in 2018 by mathematician Leland McInnes. UMAP stands for “Uniform Manifold Approximation and Projection” — if that’s a mouthful, don’t worry. (If you want an explanation, <a href="https://www.youtube.com/watch?v=6BPl81wGGP8">check this video out by Dr. Letitia</a>) In the simplest way I can explain: the UMAP algorithm tries to connect the 784D image data by dots and lines depending on how clustered the data is (kinda like a social network!); then the whole graph is <em>projected</em> using <em>topology,</em> down into a lower-dimensional space. If you want to visualize the data points, you’ll have to project all the way down into 2D or 3D space, as I’ve done here below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PRcq0EHo9Uu0-lYkqqVZ7w.jpeg" /><figcaption>Image source: Raveena Jayadev. The “Component 1” and 2 stand for UMAP Components. It’s the same thing basically as “principal components” in “PCA” for dimensionality reduction, if you’ve heard of it.</figcaption></figure><pre>mapper = umap.UMAP(n_components=2, random_state=40).fit(random_x_train)<br>umap_components = mapper.transform(random_x_train)</pre><p>The distance you see up here between clusters reflects the fact that in some concrete way, certain digits like 0 and 7 are “far away” or “pretty separate” from 7 and 9. Of course you might say “Raveena! every digit is separate” and that’s true, but when handwriting digits, sometimes pen strokes can be confusing. And that’s the underlying structure UMAP is trying to compress into 2D so we can see it for ourselves!</p><p><em>(Disclaimer: </em>Although the UMAP components show clear clustering in the scatterplot, the components axes themselves are not really human interpretable. However, the decision-making prediction process still is.)</p><p><strong>Learning the “Mental” Generative Models</strong></p><p>Now it’s time to learn the “mental models” — the generative probability models for each class of handwritten digit the computer will learn from data in order to make educated inferences at classification time. To create the actual model, we will use a technique in the <em>sci-kit learn</em> module called <em>kernel density estimation (KDE)</em>. Imagine a data set of points, some of them close to each other, some more lonely and isolated. The estimation process is visualized below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*Gx8pV4xGGxe52riKPjcoJw.jpeg" /><figcaption>Image Source: Wikipedia. The black tick marks on the bottom are data points in the set, and the red dotted lines are the little bell curves placed at each point. KDE averages the bell curves into the solid blue line to get the general shape. (Link: <a href="https://en.wikipedia.org/wiki/Kernel_density_estimation">https://en.wikipedia.org/wiki/Kernel_density_estimation</a>)</figcaption></figure><p>Once we learn the generative “mental” models for each digit, we can plot the models together and use 10 different colors to denote the digits 0–9 and their model shapes. Each “shapely blob” of color below represents the internal model the computer has learned from data for each digit, similar to how human brains have internal models for how digits should look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*WOxL7TXUXywwGJCh9yqasQ.jpeg" /><figcaption>Image credit: Raveena Jayadev</figcaption></figure><p>Now that our computer has generative models for each digit, it’s time to do…inference! What I want to emphasize in this project is two things: 1) we can program image classifier that infers and decides with internal “mental” models like a human brain can, and 2) this makes the AI system easy to <em>troubleshoot</em>. In an era where black-box deep AI systems are being deployed all over the world <a href="https://towardsdatascience.com/towards-trans-inclusive-ai-a4abe9ad4e62">and often discriminate against marginalized people</a> (article by Zachary Hay of <em>Towards Data Science</em>), it’s extremely hard to troubleshoot the discriminatory predictions those systems can make––and if you can’t troubleshoot, the system might as well be dangerous in my opinion.</p><p>Let me show you how we can check the system we’re creating in this article, though. Here’s the summarized code for how our system would infer and predict the content of an unseen test image:</p><pre>def decision(sample, mental_models):<br>... (summary) <br>   <strong>if likelihood_ratio &lt;= 3.0:</strong><br>        print(&quot;There&#39;s very little evidence that my first choice, is better than my second choice.&quot;<br>       <strong>return &quot;I refuse to decide!&quot;</strong><br>   <strong>elif (likelihood_ratio &gt; 4.0) and (likelihood_ratio &lt;= 10.0):</strong><br>        print(&quot;There is a bit of evidence that my first choice is better than my second choice&quot;)<br>       <strong>return &quot;maybe digit {} over {}&quot;<br>   else:</strong> print(&#39;There is strong evidence that my first choice, is better than my second choice.&#39;<br>       <strong>return [confident machine choice]</strong></pre><p>(The machine prediction code looks just like an <em>if-else statement</em> (as said above), and the machine can predict with <em>different</em> levels of confidence, which is pretty intuitive!)</p><p>Now suppose we gave this AI system an image of a 3, and instead it predicted a 5. How could we troubleshoot the model to see how it made a mistake? Well, follow me here:</p><ul><li>The machine decided from the if-else statement, that under its generative “mental” models, a 5 was more likely from the image data than a 3.</li><li>If-else decisions are pure logic, so that means the issue is not with the decision, but either the <em>statistical mental models</em> or the <em>feature extraction</em>. We can ask the machine to display the mental models and feature extraction, both as I did above. From there we can analyze the shortcomings of the models, and see how to adjust them.</li></ul><p>In these 2 steps, we already identified where the problems might lie in this AI system if it made a mistake, By contrast, many of the deep learning systems connected to surveillance technologies around the world cannot even be troubleshooted in this way, and that makes for potentially dangerous AI.</p><p>I hope you, the reader, have come away from this article with a hope that 1) we can create AI systems that can potentially make decisions intuitively like humans can, and 2) we can potentially <em>troubleshoot</em> the AI model if it makes mistakes, and backtrack the error all the way into the internal parts of the system and understand what to fix and what to scrap.</p><p>Now admittedly, you might be telling me: “Raveena, the “UMAP” feature extraction doesn’t seem intuitive!” Yes, I know. The features are unfortunately not human-interpretable. However, there is a way to modify this system so that it <em>can</em> be interpretable, albeit in a different way. If you’re interested to know, <a href="https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science.pdf">this article in 2015 by MIT Prof. Joshua Tenenbaum</a><em> </em>talks about making a AI handwritten character classifier system using internal “mental models” that are quite similar<em> </em>to our probabilistic generative models. In fact, it’s so intuitive and data efficient, it only learns from <em>30 data points</em> and imitates human handwritten symbols eerily well. In my opinion, it’s one of the most revolutionary creations for the future of AI.</p><p>By the way, <a href="https://github.com/galaxyenby1997/Image-Classifier-generative">here’s a link to my code</a> if you want to see the full implementation details: my Github is public and free to access :)</p><p><strong>References</strong></p><ol><li>Thomas Parr, Karl Friston, The Anatomy of Inference: Generative Models and Brain Structure (2018), <em>Frontiers in Computational Neuroscience</em>,<em> </em>2018.</li><li>S.R. Kheradpisheh et al., Deep Networks Can Resemble Human Feed-forward Vision in Invariant-Object Recognition (2016), <em>Nature</em>.</li><li>Joshua Tenenbaum, Brendan Lake, Human-Level Concept Learning through Probabilistic Program Induction (2015), <em>Science</em>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=266bd657aa4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/programming-an-intuitive-image-classifier-part-1-266bd657aa4">Programming An Intuitive Image Classifier, Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>