<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by saeed Tech Advisor on Medium]]></title>
        <description><![CDATA[Stories by saeed Tech Advisor on Medium]]></description>
        <link>https://medium.com/@mskhan333?source=rss-a16a969b7cbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jhGsFYRZkPWhgE7iq7i2KA@2x.jpeg</url>
            <title>Stories by saeed Tech Advisor on Medium</title>
            <link>https://medium.com/@mskhan333?source=rss-a16a969b7cbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:38:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mskhan333/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[And we get… perfect results! Wow.]]></title>
            <link>https://medium.com/@mskhan333/and-we-get-perfect-results-wow-8111153675f6?source=rss-a16a969b7cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/8111153675f6</guid>
            <dc:creator><![CDATA[saeed Tech Advisor]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 02:57:56 GMT</pubDate>
            <atom:updated>2023-06-13T02:57:56.128Z</atom:updated>
            <content:encoded><![CDATA[<p>And we get… perfect results! Wow.</p><p>When you observe perfect performance on a machine learning task, you should be very worried indeed. Just as the previous article warned you against having the wrong attitude towards 0%, it’s a good habit to smell a rat whenever you see perfect results. An expert’s first thought would be that either the test was much too easy or the labels leaked or the data weren’t as pristine as we hoped. Exactly the same thoughts a seasoned college professor would have upon discovering that every student answered every question flawlessly. They’d be furiously debugging instead of patting themselves on the back for good teaching.</p><p>Since 100% isn’t a realistic result for a lifelike scenario, let’s imagine we got some other high-but-not-too-high score, say 98%.</p><p>Great performance, right? Sounds like great performance in pristine data to boot. With all the statistical hypothesis testing bells and whistles taken care of! We’ve built an awesome Tesla/Huxley detector, let’s launch it!!!</p><p>Watch out.</p><p>You’ll be tempted to jump to conclusions. This might not be a Tes/Hux detector at all, even if it gives great results in testing.</p><p>The mistake that experts make has less to do with the technical and more to do with the human side of things: giving in to wishful thinking.</p><p>Experts aren’t immune to the mistake of wishful thinking.</p><p>Don’t assume these results mean anything at all about what the system is actually detecting. We have no evidence that it has learned – or worse, that “it understands” – how to tell Hux and Tes apart. Look closer…</p><p>Did you notice that little detail that’s present in the background of all photos of Tes? Whoops, turns out we accidentally built a radiator / not-radiator classifier!</p><p>It’s a very human mistake to over-focus on the parts of the problem and data that are interesting to us – like the identity of the two cats – and snooze past the other bits, like the presence of the radiator in most of the photos of Tes since that’s where she likes to hang out and therefore it’s also where she’s most likely to be photographed. Unfortunately, she loves the radiator so much that the same issue turns up in both the training data and the test data, so you won’t catch the problem by testing on data from the same population you used for training.</p><p>All this is (somewhat) okay as long as you (and your stakeholders and users) have the steely discipline of:</p><p>Not reading cute stories into the results, like “This is a Hux/Tes classifier.” This kind of language implies something far more general than we have evidence for. You’re probably doing it wrong if your understanding of what a machine learning system is actually doing fits into a pithy newspaper headline… It should, at a minimum, bore you to death with its details.</p><p>Staying painfully aware that if there’s a difference between the setting you tested your system in and the setting you’re training it in, you have no guarantee that your solution will work.</p><p>This kind of discipline is rare, so until we have better data literacy, all I can do is dream big and do my bit for educating people.</p><p>This system will only work with photos taken in the same manner as the testing and training datasets – photos from the same apartment in the winter, since that’s when Tesla tended to hang out by the radiator while Huxley didn’t. If you always take photos in the same way and you had a lot of them, who cares how your system does the trick of assigning labels as long as it gets you the right ones. If it uses the radiator to win at the labeling game, that’s fine… as long as you don’t expect it to work in any other context (it won’t). For that, you need better, broader training and testing datasets. And if you move the system to a different apartment, even if you think you understand how your system works, test it afresh anyway to ensure it still works.</p><p>The world represented by your data is the only one you’re going to succeed in.</p><p>Remember, the world represented by your data is the only one you’re going to succeed in. So you need to think carefully about the data you’re using.</p><p>Avoiding the expert mistake</p><p>Never assume you understand what an AI system is doing. Believing that your simple understanding of a complex thing is bulletproof is the height of arrogance. You’ll be punished for it.</p><p>Never assume you understand what an AI system is doing. Believing that your simple understanding of a complex thing is bulletproof is the height of arrogance. You’ll be punished for it.</p><p>It’s easy to miss real-world subtleties in your data. No matter how talented you are at it, if you work long enough in data science, you’ll almost surely make this wishful-thinking mistake at least once until you learn the hard way to severely curtail your conclusions – the best habit you can build is to avoid reading extra meaning into things until you’ve thoroughly checked the evidence. You’ll know you’ve arrived at the next level when your inner monologue about your results sounds less like a TED talk and more like the fine-print of the world’s most boring contract. This is a good thing. You can dazzle with your fast talk later if you insist, just don’t embarrass data literacy by believing your own hyperbolae.</p><p>The only thing you can learn from good testing results is that the system works well in environments, situations, datasets, and populations that are similar to the testing conditions. Any guesses you’d be tempted to make about its performance outside those conditions are fiction. Test carefully and don’t jump to conclusions!</p><p>Thanks for reading! How about a YouTube course?</p><p>If you had fun here and you’re looking for an applied AI course designed to be fun for beginners and experts alike, here’s one I made for your amusement:</p><p>Enjoy the entire course playlist here: bit.ly/machinefriend</p><p>Liked the author? Connect with Cassie Kozyrkov</p><p>Let’s be friends! You can find me on Twitter, YouTube, Substack, and LinkedIn. Interested in having me speak at your event? Use this form to get in touch.</p><p>Interested in trying out image classification?</p><p>Get started with Google Cloud Vision API and AutoML Vision here:</p><p>Vision AI | Derive Image Insights via ML | Cloud Vision API | Google Cloud</p><p>Today, meet tomorrow. Join us October 11 – 13 for Google Cloud Next. Register now Derive insights from images with AutoML…</p><p>bit.ly</p><p>Data Science</p><p>Editors Pick</p><p>Machine Learning</p><p>Artificial Intelligence</p><p>Technology</p><p>528</p><p>5</p><p>Cassie Kozyrkov</p><p>Towards Data Science</p><p>Written by Cassie Kozyrkov</p><p>153K Followers</p><p>·</p><p>Writer for</p><p>Towards Data Science</p><p>Chief Decision Scientist, Google. ❤️ Stats, ML/AI, data, puns, art, theatre, decision science. All views are my own. twitter.com/quaesita</p><p>Follow</p><p>More from Cassie Kozyrkov and Towards Data Science</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>in</p><p>Towards Data Science</p><p>The Best Learning Paths for AI and Data Leadership</p><p>How to muscle up on data-related topics quickly</p><p>·</p><p>7 min read</p><p>·</p><p>Apr 30</p><p>1.4K</p><p>11</p><p>Jacob Marks, Ph.D.</p><p>Jacob Marks, Ph.D.</p><p>in</p><p>Towards Data Science</p><p>How I Turned My Company’s Docs into a Searchable Database with OpenAI</p><p>And how you can do the same with your docs</p><p>15 min read</p><p>·</p><p>Apr 25</p><p>3.8K</p><p>48</p><p>Leonie Monigatti</p><p>Leonie Monigatti</p><p>in</p><p>Towards Data Science</p><p>10 Exciting Project Ideas Using Large Language Models (LLMs) for Your Portfolio</p><p>Learn how to build apps and showcase your skills with large language models (LLMs). Get started today!</p><p>·</p><p>11 min read</p><p>·</p><p>May 15</p><p>1.2K</p><p>7</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>Why generative AI is a raw material, not a finished product</p><p>The new rhetoric around AI regulation</p><p>·</p><p>3 min read</p><p>·</p><p>May 21</p><p>497</p><p>10</p><p>See all from Cassie Kozyrkov</p><p>See all from Towards Data Science</p><p>Recommended from Medium</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>The best New Year’s resolutions you can make</p><p>Resolutions that actually work, according to a decision scientist</p><p>·</p><p>9 min read</p><p>·</p><p>Dec 28, 2022</p><p>2.6K</p><p>42</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>Are your goals hurting or helping you?</p><p>How to set better goals without shooting yourself in the foot</p><p>·</p><p>8 min read</p><p>·</p><p>Jan 4</p><p>915</p><p>9</p><p>Lists</p><p>What is ChatGPT?</p><p>9 stories</p><p>·</p><p>94 saves</p><p>Stories to Help You Grow as a Designer</p><p>11 stories</p><p>·</p><p>70 saves</p><p>Good Product Thinking</p><p>11 stories</p><p>·</p><p>118 saves</p><p>Stories to Help You Grow as a Software Developer</p><p>19 stories</p><p>·</p><p>112 saves</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>in</p><p>Towards Data Science</p><p>Why is Mean Squared Error (MSE) So Popular?</p><p>A quick look at everyone’s favorite loss function</p><p>·</p><p>4 min read</p><p>·</p><p>Nov 19, 2022</p><p>507</p><p>6</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>Break the cycle of bad self-improvement advice</p><p>How motivational gurus prey on the vulnerable</p><p>·</p><p>6 min read</p><p>·</p><p>Jan 1</p><p>2.1K</p><p>26</p><p>Cassie Kozyrkov</p><p>Cassie Kozyrkov</p><p>in</p><p>Towards Data Science</p><p>Cynicism: a must-have for data scientists</p><p>The reason that asking “why?” won’t always lead to impact</p><p>·</p><p>6 min read</p><p>·</p><p>Sep 22, 2022</p><p>720</p><p>4</p><p>Madison Hunter</p><p>Madison Hunter</p><p>in</p><p>Towards Data Science</p><p>How to Use Micro-Credentials to Get Your Foot in the Door as a New Data Scientist</p><p>Micro-credentials are a place to start in data science if you know how to leverage them properly</p><p>·</p><p>8 min read</p><p>·</p><p>Jan 25</p><p>95</p><p>2</p><p>See more recommendations</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8111153675f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>