<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dominion on Medium]]></title>
        <description><![CDATA[Stories by Dominion on Medium]]></description>
        <link>https://medium.com/@dominionakinrotimi?source=rss-a035a7a35e11------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1MozeeIwvws1d4awGfWwwQ.png</url>
            <title>Stories by Dominion on Medium</title>
            <link>https://medium.com/@dominionakinrotimi?source=rss-a035a7a35e11------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:43:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dominionakinrotimi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Ethics: Data Teaches a Model to Discriminate]]></title>
            <link>https://dominionakinrotimi.medium.com/data-ethics-data-teaches-a-model-to-discriminate-2c6cc7538ba7?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c6cc7538ba7</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-ethics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 15:54:50 GMT</pubDate>
            <atom:updated>2026-11-06T15:54:50.270Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Diaries. Episode 7</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-38GhSpGovVJhLcJSwlHw.jpeg" /></figure><h4>Introduction to Data Ethics:</h4><p>“What harm could this system cause, even unintentionally?”</p><p>Data ethics is the practice of asking that question throughout the entire data lifecycle, from collection to deployment and beyond.</p><p>It is the discipline of seeing data not as abstract numbers but as fragments of human lives. It means recognizing that our models do more than make predictions: they can influence decisions that ripple through society. Decisions that people may trust, depend on, or be harmed by.</p><p>Let’s get into the practical side of it.</p><h3>Historical data teaches a model to discriminate</h3><p>Imagine you are training a machine learning model to screen job applicants. You have obtained historical hiring data. You clean it, engineer features and build a classifier to rank new candidates.</p><p>You have completed the technical work, but is the historical data a fair representation of the people you want to hire?</p><blockquote>Historical hiring data is not automatically ethically neutral.</blockquote><p>It can preserve the effects of earlier decisions: which universities recruiters preferred, which names or backgrounds were familiar to them, and which candidates were repeatedly given opportunities. A model trained on those patterns will definitely learn to reproduce them.</p><p>So now, instead of identifying the most suitable candidates, the model may learn to identify candidates who resemble people the organization hired in the past. That is how a system can automate an old pattern of discrimination and make it look like objective, data-driven decision-making. lol!</p><p>This is not a hypothetical concern because Reuters reported in 2018 that Amazon abandoned an experimental AI recruiting tool after it was found to disadvantage women. The system had learned from resumes submitted over a ten-year period, most of which came from men. It penalized resumes containing the word “women’s” and downgraded graduates of two women’s colleges. The company could not be confident that removing those specific signals would prevent the system from finding other ways to discriminate.</p><blockquote>Are you beginning to question the assumption that past data is always a valid template for the future? Well, you should</blockquote><blockquote>It is true that we cannot build the future without learning from the past but we must develop the habit of examining our datasets and asking whether they are appropriate, representative, and ethically fit for the decisions we want to make.</blockquote><blockquote>— DominionAkinrotimi</blockquote><p>THIS is “data ethics.&quot;</p><p>Now you know that our routine technical task of feeding data into an algorithm is not morally neutral.</p><h3>Where does ethics fit into the machine learning lifecycle?</h3><p>One thing I want us to understand is that data ethics is that it is a thread running through the entire machine learning lifecycle. Every stage always presents a different ethical question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3DF65FGi8C8GdXUPGZONOQ.png" /><figcaption><em>Figure 1. Ethical review is continuous: each stage has its own questions, and monitoring can send us back to earlier decisions.</em></figcaption></figure><ol><li>Problem definition: Are we solving the right problem, and who decided it was worth solving?</li><li>Data collection: Do we have a legitimate reason to collect and use this information?</li><li>Data preparation: Are we removing, mislabelling, or underrepresenting particular groups?</li><li>Exploratory analysis: What patterns, gaps and imbalances are present in the data?</li><li>Feature engineering: Could a seemingly ordinary feature act as a proxy for a protected characteristic?</li><li>Model development: What assumptions and historical patterns is the model learning?</li><li>Evaluation: Does the model perform reliably across the groups affected by it?</li><li>Deployment: Who will rely on its outputs, and what human oversight is available?</li><li>Monitoring: Are new biases, errors or harms emerging as people and circumstances change?</li></ol><p>Building a model is way more than “Does it work accurately?”</p><p>Who does it work for?</p><p>Who does it disadvantage?</p><p>What happens when people rely on it?</p><blockquote>Now you know</blockquote><p>In the next article we will uncover and explore the four pillars of data ethics.</p><h3>A few sources for further reading</h3><p>· Reuters (2018), “Amazon scraps secret AI recruiting tool that showed bias against women.” <a href="https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G">https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G</a></p><p>· A Review on Fairness in Machine Learning, ACM Computing Surveys. <a href="https://doi.org/10.1145/3494672">https://doi.org/10.1145/3494672</a></p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c6cc7538ba7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a Data Scientist Sees When You Post a Photo]]></title>
            <link>https://dominionakinrotimi.medium.com/what-a-data-scientist-sees-when-you-post-a-photo-c7312f0cbc2e?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7312f0cbc2e</guid>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 11:11:15 GMT</pubDate>
            <atom:updated>2026-04-01T11:11:15.333Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Diaries. Episode 6</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cSbGyIvebT6U-91hqigPIg.png" /></figure><blockquote>Disclaimer: This article is a little bit lengthy but I promise that by the end of the 11 minutes you devote to learning by reading this, you would have develop a new mindset towards sharing digital data. Looking forward to your feedback on this.</blockquote><h3>Introduction</h3><p>A few years ago, I noticed something that made me very uncomfortable.</p><p>I went to visit a friend with another friend of mine and we were in the room talking about how she was almost kidnapped few days ago. We chatted for a while, and a few minutes later, I opened WhatsApp. And there it was. Her sharing unnecessary and sensitive details again on her status.</p><p>I’m not one to judge and yes, I didn’t point it out right there but she always posts her location on her whatsapp status. “On my way to Benin”, takes a picture as soon as she arrives her destination and post, “going to school this Saturday not prepared for the stress”,</p><p>…. gurllll??!!!!</p><p>Mind you, she isn’t stupid… she is a very smart young woman. So I didn’t know what excuse to even make for her.</p><p>I just couldn’t help but judge her actions in as much as I felt so terrible and scared for her: “If anyone wanted to kidnap you, you have literally provided everything they need dear”</p><p>Later on, I stylishly asked her why she does that.</p><p>And guess what, her reasons were so valid.</p><p>One of them being, anxiety, she scared and wants her people to always know her whereabout, that’s why she’s always posted stuff like that.</p><p>Now I’m not saying… “Because she posted, that is why she …”</p><p>This article is to tell you what no one had told her about how these things add up.</p><p>And in all sincerity, until that moment, I hadn’t really thought about it either.</p><h3>Why do you do what you do?</h3><p>We all share things online constantly but have we ever stopped to ask what we’re actually doing and why we are doing that?</p><p>Why do you share what you share? Before I make any post or send anything I always have a why</p><p>What is your “why”?</p><p>For some, it is to show people you are having a nice time<br>For engagement. Just for posting sake. To build their brand, etc.</p><p>These are normal things. The kind of things everyone does.</p><p>But every one of those actions is a piece of information.</p><p>And once combined, they start to form something none of us intended.</p><h3>A Case Study of My Brain</h3><p>Let me give you an example from my own life.</p><p>There was another time. Back in my campus fellowship days.</p><p>The lead usher didn’t show up for church one Sunday. Someone said she was sick. So, a few of us decided to go visit her after service but nobody knew her room. Someone knew her hostel. So we went.</p><p>I had never been to her place before. When we got to the environment, I led everyone straight to her door. Didn’t ask for direction.</p><p>The exact room.</p><p>When she opened the door, she was surprised and asked us. “How did you find my room?”</p><p>I thought about it for a second. Then I realized.</p><p>She posted pictures constantly. Photos from her window. Photos in front of her apartment. Photos of her street. The inscription on her wall. Videos of her walking out of her room. The color of the gate.</p><p>I had seen these images dozens of times over months. Just scrolling. But my brain did what brains do. It stored them. Stitched them together. Built a map.</p><p>I wasn’t a stalker. I wasn’t a hacker. I wasn’t a spy.</p><p>She had been sharing her life. But she hadn’t realized she was also sharing her location. Her routine. Her doorstep.</p><h3>Case Explained</h3><p>Few months later I joined a class and learned what was happening under the surface that neither of us understood at the time.</p><p>Every photo she posted carried something called <strong>EXIF data</strong>, just a fancy acronym for the information your phone quietly stamps onto every image you take. Full meaning: <strong>Exchangeable Image File Format</strong>.</p><blockquote>It’s a standard used by every smartphone manufacturer.</blockquote><blockquote>When she took a picture, her phone embedded GPS coordinates, precise timestamp, camera settings, and sometimes even a unique device identifier.</blockquote><blockquote>Instagram normally strips most of that when you upload.</blockquote><p>But not the image itself. The visual details like the buildings around, the gate, the colors,</p><p>those are visual metadata and no platform can strip what the picture is actually a picture of. Unless, let’s say, you paint the entire picture white. 😂</p><p>My brain took those visual details and performed something called <strong>spatial triangulation</strong>.</p><p>It’s what humans do unconsciously.</p><p>You see the same window from three different angles over three months, and your brain calculates where that window faces.</p><p>You see the same doorframe in morning light and evening light, and your brain knows which direction the apartment faces.</p><p>You see the same street pattern repeated, and your brain builds a route.</p><p>Now, this isn’t applicable to everyone but if you are like me, I know you can relate to this.</p><p>I didn’t do anything special but she had fed my brain enough data over time for it to complete the map.</p><h3>The Brain X AI</h3><p>Modern computer vision systems work the same way my brain handled that info.</p><p>When you train an image recognition model, you don’t give it one photo of a door but hundreds and thousands. <br>The model doesn’t memorize the images.</p><p>It extracts features like edges, textures, color distributions, spatial relationships</p><p>and builds a mathematical representation of what a door looks like. Then it can recognize that door in a new image it has never seen before.</p><p>My brain did the same thing with her apartment.</p><p>I didn’t have thousands of images. But I had enough.</p><p>Now, she wasn’t hacked neither was her data stolen. She gave it freely on her own volition, one post at a time. And I used it exactly the way systems are designed to use data: to build a model that predicts where something is.</p><h3>So What?</h3><p>We share things online constantly, but we almost never stop to ask what we’re actually doing.</p><p>You post a photo of your breakfast. You check in at a coffee shop or eatery. You share a story about your child’s event at school. You tweet something frustrating about your commute.</p><p>These are normal things. The kind of things everyone does.</p><p>But every one of those actions is a piece of information.</p><p>On an iPhone, go to any photo, swipe up, and you’ll see the location, the time, sometimes even the altitude.</p><p>On Android, it’s in the photo details. That metadata is stored whether you see it or not. When you upload to some platforms, that metadata stays attached. When you upload to others, it’s stripped. But what about the visual metadata ? the actual content of the image which can never be stripped. That’s what the image is.</p><p>So when you post that photo, you’re not just sharing a memory. You’re sharing coordinates or visual coordinates. Timestamps or visual timelines. To a person, it’s a picture. To a system or to another human brain with enough exposure, it is geolocation data rendered as pixels.</p><p>I’m not saying this to scare you.</p><p>I still post photos and I still use apps. But I now know what I’m actually doing when I hit “share.” And that knowledge changed how I move.</p><h3>An Erroneous Belief of Mine.</h3><p>I used to think my phone was listening to me. I can bet that you know the experience but I won’t. <br>When you mention something aloud…a product, a place, a topic</p><p>and boom it appears in your feed when you go online. It feels invasive, like someone is watching me.</p><h4>What is really happening:</h4><p>Your phone has a microphone. It could listen. But that would be inefficient. Audio data is large, processing it requires power and transmitting it requires bandwidth and more importantly, companies don’t need to.</p><p>What they have is something called <strong>behavioral profiling</strong>.</p><p>Every app you use tracks what’s called <strong>telemetry</strong></p><p>Telemetry is not the content of what you do, but the fact that you did it.</p><p>What time you opened the app. How long you stayed. What you scrolled past. What you paused on. What you clicked. What you typed and then deleted.</p><p>These are called engagement signals. These little signals feed into recommendation algorithms that use collaborative filtering ,</p><blockquote>Collaborative filtering is a mathematical technique that finds patterns across millions of users.</blockquote><blockquote>Simply put they look for people who move like you move. If a thousand people with your habits searched for a product, the algorithm assumes you might want it too</blockquote><p>Mathematically, to the system, you look like the type of person who wants it.</p><p>It still feels creepy. I still get sus. but understanding the difference between surveillance vs. pattern matching was how I went from: “this is creepy and I don’t know why”</p><p>to “this is how the system works, and I can decide how I want to engage with it”.</p><h3>Permissions</h3><p>Permissions are another thing I so used to ignore.</p><p>I remember downloading an app to identify songs. I won’t mention the name. But you know it.</p><p>It worked well. I used it for weeks. Then one day I noticed something. My microphone indicator was on. Even when I wasn’t using the app. Even when my phone was sitting on my desk. I straight up deleted it.</p><blockquote>That app requested microphone permission. I granted it.</blockquote><blockquote>On both iOS and Android, once you grant a permission and say “yes, I do”, that app can access that sensor whenever it’s running in the background unless you go out of your way to restrict it and most people do not even try to.</blockquote><p>I don’t know if the app was doing anything malicious.. It might have been listening for music detection in the background. Or it might have been keeping the microphone active so it could respond faster when opened. Either way, it had access I didn’t intend to give.</p><p>I didn’t know exactly what was happening. But I knew enough to understand that an app that identifies songs didn’t need access to my microphone when I wasn’t using it.</p><p>I deleted it oh. I had given it permission without thinking, and I didn’t like what that permission actually meant. I wasn’t comfortable with it.</p><p>Since then, I’ve started paying attention to what apps ask for. A flashlight app doesn’t need my contacts. What for to show them the light at the end of the tunnel??? A calculator doesn’t need my camera.</p><p>Once you start taking cognizance of these presumedly little things, you realize how many apps ask for things they have no legitimate reason to need. And most of us just click “allow” without a second thought.</p><h3>On Posting Other People’s Children</h3><p>Posting someone else’s child online because they’re cute? That’s not your decision to make. That child didn’t consent. Their parent didn’t consent.</p><p>There are people out there who take innocent photos of children and use AI to generate nude images. Let that sink in. The photo you thought was harmless could end up anywhere, doing anything.</p><p>Once that image leaves your phone, you have no control over it. No idea who saves it, where it goes, or what they do with it.</p><p>Ask the parents before you post. Or better yet, suppress the urge and just don’t.</p><h3>Why Should You Care About This?</h3><p>You might be reading this and thinking, <em>So what? I have nothing to hide.</em></p><p>This was me few years ago. I’ll love you to understand that:</p><blockquote>Privacy is about choosing not “hiding things”.</blockquote><blockquote>It’s about maintaining the right to decide what parts of your life become data, who gets to see them, and what they get to do with that information.</blockquote><blockquote>You lock your front door not because you have something to hide, but because your home is yours. You close the bathroom door not because you’re doing something shameful, but because some moments aren’t for public consumption.</blockquote><blockquote>Your digital life deserves the same respect.</blockquote><p>The people who collect your data aren’t doing it because they find you personally interesting.</p><p>They’re doing it because your patterns are valuable.</p><p>Your attention can be sold.</p><p>Your habits can be predicted.</p><p>Your decisions can be gently guided toward outcomes that serve someone else’s bottom line.</p><p>I’m not saying there’s a villain in a dark room watching your every move. Most of this is automated, innocent systems and softwares doing what they were built to do and it almost makes it worse because when there’s no human in the loop, there’s also no human asking <em>should we be doing this?</em></p><p>The question isn’t whether you have something to hide. The question is whether you’re comfortable with your life being turned into a product without your explicit, informed consent.</p><p>For me, the answer was no. Not to and for everything, obviously.</p><h3>What You Can Actually Do About It</h3><p>I’m not going to tell you to delete all your social media and move to a cabin in the woods. Here are the things that actually made a difference for me.</p><h4>1. Review your permissions. Right now.</h4><p>Go to your phone settings. Look at which apps have access to your microphone, camera, location, and contacts.</p><p>On iPhone: Settings &gt; Privacy &amp; Security. On Android: Settings &gt; Privacy &gt; Permission Manager.</p><h4>2. Start noticing what your posts actually contain</h4><p>Before you hit share, take one extra second. Look at the photo you’re about to post. What’s in the background? Is your house number visible? Your license plate? Your child’s school uniform? A street sign near your home?</p><h4>3. Turn off location tagging for photos by default</h4><p>On both iPhone and Android, you can set your camera to not embed location data in photos. If you now want to genuinely share your location for a specific post, you can turn it on for that moment.</p><h4>4. Ask yourself the two-question test</h4><p>Before you post anything, ask yourself two things:</p><ul><li><em>“Would I be comfortable with a stranger knowing this about me?”</em></li><li><em>“Would I be comfortable with this information existing permanently, tied to my identity, available to whoever has access?”</em></li></ul><p>If the answer to either is no, maybe it’s best that post stays in your drafts or becomes a private message to the people who actually need to know.</p><h4>5. Pay attention to what you’re paying with</h4><p>Every free app has a cost. When you sign up for something, take a second to think: “<em>What am I actually paying here?”</em></p><h4>6. Have the conversation</h4><p>Talk to your people about this. Share what you’ve learned. Ask them what they think. I have made you see things differently. You might be that person for someone else.</p><p>This stuff isn’t intuitive. No one teaches it. We all learned by sharing first and understanding later.</p><h3>Concluding with Data sounds Ironic</h3><p>Data is the beginning of information. The genesis and foundational requirement to building anything useful.</p><p>You most likely thought data is something that belonged to “tech people.” Data science. Algorithms. AI. Things happening in places you didn’t work and didn’t fully understand.</p><p>But I believe by now, you should see it differently.</p><p>Data is not abstract. Data is what you and I actually do. The places you go. The things you search. The posts you linger on. The things you buy. The people you interact with.</p><p>That’s data and it’s being collected, stored, and used whether you think about it or not.</p><p>Every free app you use has a business model. Servers cost money. Engineers cost money. If you are not paying with your wallet, you are most likely paying with something else. Usually, it’s your data or your attention.</p><p>I’m not saying this to make you delete everything. As I said, I still share things online, still use free apps but I’ve stopped pretending there’s no cost. There is a cost. And I want to know what I’m paying.</p><p>What does this app actually collect?<br>Who has access to it?<br>What can I do about it?</p><p>Don’t see permissions as annoying pop ups, look at it, read it before you default to “yes” or “I accept”, what are you accepting???</p><p>These are requests for access to your life.</p><p>Pay attention to what you post, ask yourself if the post reveals more than you mean to reveal?</p><p>Know the difference between security and privacy.</p><blockquote>Security is about keeping people out.</blockquote><blockquote>Privacy is about choosing what to share and with whom.</blockquote><blockquote>You can have strong security and poor privacy.</blockquote><p>Now you know.</p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7312f0cbc2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Problem Framing: The Most Important (and Skipped) Step in Any Data Project]]></title>
            <link>https://dominionakinrotimi.medium.com/problem-framing-the-most-important-and-skipped-step-in-any-data-project-6b720881bf1d?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b720881bf1d</guid>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 19:43:47 GMT</pubDate>
            <atom:updated>2026-02-06T19:43:47.056Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Diaries. Episode 5</p><p><em>Problem framing is the disciplined process of translating a vague business headache into a concrete, measurable, and solvable machine learning or analytics objective. Without it, you risk building a technically perfect solution to the wrong problem.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i1WK07S2K7FUurSy9tEZ8g.jpeg" /><figcaption>Background image generated by AI on Freepik</figcaption></figure><p>Essentially, it is the act of converting ambiguity into a structured form that can be addressed with data.</p><p>This does not just apply to data, AI, ML, ... It is applicable all around. Let’s say you want to build a house, you wouldn’t start hammering nails because someone said, “I need shelter.” You would ask questions like, “How many people?” “What climate?” “What is your budget?” “What is the set timeline?” ... Only then do you choose materials and methods.</p><p>The mechanism has three interlocking gears:</p><ol><li><strong>Input/Output Definition:</strong> This is our functional requirement. What <em>observable data</em> can we use as input? Note: relevant input. Then, what <em>actionable output</em> must we produce that solves the problem? The output might be a churn-risk score, a binary prediction, weekly demand forecast to optimize inventory or a ranked list of customers most likely to cancel.</li><li><strong>Success Metric Alignment:</strong> What does “better” mean? Does the model’s accuracy metric actually move the business needle (e.g., reduced customer churn)? This is categorized under business goals and non-functional requirements.</li><li><strong>Constraint Acknowledgment:</strong> Assumptions, limitations, compliance sections, … What are the real-world limits? Latency? Multilingual support? Regulatory constraints? Must predictions come in under 100 ms? Cost? Infrastructure? Services/stack being used?</li></ol><p>These are simple software development practices. Now, note that problem framing is not a meeting, but it is a thinking process that ensures that your SDLC SRS or BRD is meaningful. If you skip problem framing, your SRS is at a high risk of consisting of features without clear business alignment.</p><h3>How Does Problem Framing Actually Save Time?</h3><p>Most data projects fail because teams jump straight to solving the presented symptom without diagnosing the root cause.</p><p>Stakeholder presents symptom: “Customers complain about late deliveries!”</p><p>The team immediately builds a complex LSTM model predicting shipment delays.</p><p>Three months later, the model is 95% accurate, but complaints continue.</p><p>Meanwhile, the real issue was maybe the warehouse picking system or the shipping partner is unreliable, problems that no prediction model will solve.</p><blockquote>One vital point to note is that not every “problem” you see is the actual root cause. Sometimes it’s just the visible symptom of something else or something deeper.</blockquote><h3>Why should framing come first?</h3><p>Framing your problem doesn’t just provide clarity by constraining and guiding every technical decision that follows. It defines what “data quality” means for this specific project. It also dictates the correct model evaluation metric, and it determines the necessary deployment architecture (real-time API vs. batch job).</p><p>I’ll be sharing the questions that expose gaps in your understanding of a problem in a later post.</p><h3>Conclusion</h3><p>The most expensive waste in data work is efficient, elegant, technically sophisticated work solving an irrelevant problem. The purpose of this vital step is to be WRONG early, not RIGHT late.</p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b720881bf1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Cleaning Data Like a Pro: Checking Your Data for Errors]]></title>
            <link>https://dominionakinrotimi.medium.com/a-practical-guide-to-cleaning-data-like-a-pro-checking-your-data-for-errors-d7cfad148cff?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7cfad148cff</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[excel]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Mon, 17 Nov 2025 04:12:44 GMT</pubDate>
            <atom:updated>2025-11-17T04:12:44.557Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Diaries. Episode 4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OeisqUqzejTnSPS74GY_yw.jpeg" /><figcaption>Background image in cover image by <a href="https://www.freepik.com/author/dcstudio">DC Studio</a></figcaption></figure><p>When working with data, one small mistake can skew your entire analysis. Before going into creating dashboards, models, or reports, it’s crucial to check your data for errors. This guide walks you through duplicate detection, anomaly detection, and consistency checks using Python, Excel, and SQL. Don’t mention it, you are welcome.</p><h3>1. Duplicate Detection</h3><p>Duplicates are one of the most common sources of errors in data. They can inflate totals, mislead insights, distort trends, and even break predictive models. Before you analyze your data, it’s essential to identify and handle duplicates.</p><p>But note that not every duplicate row is an error. Sometimes, certain fields define uniqueness, and duplicates in other fields might be valid. Understanding which columns matter is key.</p><h3>Common Oversights</h3><ol><li><strong>Partial duplicates: </strong>Some rows may look different but are duplicates in the columns that matter (e.g., the same customer ordering the same product on the same date but with slightly different IDs or typos).</li><li><strong>Case sensitivity: </strong>“john@example.com” vs “John@example.com” may appear different but are the same entity.</li><li><strong>Trailing spaces or formatting: </strong>Extra spaces or different date formats can hide duplicates.</li><li><strong>Assuming the entire row defines uniqueness</strong>: Often, only a subset of columns matters for identifying true duplicates.</li></ol><h3>Defining Uniqueness</h3><p>Now, how do we define uniqueness?</p><p>Think of which columns actually make a record unique. For example, taking a look at the sales dataset below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*kI02J2rt1Q8BpyiycG08sQ.png" /><figcaption>sample sales dataset</figcaption></figure><p><strong>Full row duplicates</strong>: Row 1 and Row 2 are complete duplicates (they are the same thing) and are clearly redundant. so, we remove… one of the two.</p><p>For <strong>Subset duplicates:</strong> If you only care about <strong>CustomerID + OrderDate</strong>, Rows 1, 2, and 3 might be considered duplicates in terms of customer activity on that day, even though products differ.</p><p>The choice of columns now depends on your analysis goal:</p><p><strong>Scenario 1: Total revenue by customer</strong><br> You want to calculate how much each customer spent.</p><ul><li>If you drop duplicates using only: CustomerID + OrderDate, you <strong>lose the P002 order for customer 101</strong>, which is valid.</li><li>Instead, include ProductID as well: CustomerID + OrderDate + ProductID.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKngbkTNyvew2TYRq9-Gmw.png" /></figure><p>After removing duplicates using this subset, all valid orders are preserved:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*HvURvkluMwduc1ANFfmLNw.png" /></figure><p><strong>Scenario 2: Count of customers ordering per day</strong><br> You only care if a customer made any order that day, not which products:</p><ul><li>Using CustomerID + OrderDate is enough. Duplicate product entries for the same day are ignored.</li></ul><p>Spot the new table below after removing duplicates with CustomerID + OrderDate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*ifAUdJ3QX8qA7v9wjCdK-Q.png" /></figure><h3>Removing Duplicates using Python (pandas)</h3><pre>import pandas as pd<br><br># Load data<br>df = pd.read_csv(&quot;sales_data.csv&quot;)<br><br># Check for full-row duplicates<br>full_duplicates = df[df.duplicated()]<br>print(f&quot;Full-row duplicates: {full_duplicates.shape[0]}&quot;)<br><br># Remove full-row duplicates<br>df_clean = df.drop_duplicates()<br><br># Check for duplicates based on specific columns (e.g., CustomerID + OrderDate)<br>subset_duplicates = df[df.duplicated(subset=[&#39;CustomerID&#39;, &#39;OrderDate&#39;])]<br>print(f&quot;Duplicates based on CustomerID + OrderDate: {subset_duplicates.shape[0]}&quot;)</pre><h3>Removing Duplicates using Excel</h3><p>I already showed us a bit of it above. For context purpose, I used WPS sheet for the practical illustrations. They all work very well on excel also.</p><p>Now, to the guide:</p><ol><li>Select your dataset.</li><li>Go to <strong>Data → Remove Duplicates</strong>.</li><li>Choose columns that define uniqueness.</li></ol><p>Example: If you want one order per customer per day, select <strong>CustomerID</strong> and <strong>OrderDate</strong> only as seen above.</p><p>Excel will highlight and remove duplicates based on the columns you selected.</p><h3>Removing Duplicates using SQL</h3><pre>-- Find duplicate customer orders (CustomerID + OrderDate)<br>SELECT CustomerID, OrderDate, COUNT(*) AS duplicate_count<br>FROM Orders<br>GROUP BY CustomerID, OrderDate<br>HAVING COUNT(*) &gt; 1;</pre><blockquote><strong><em>Tip:</em></strong><em> Before removing duplicates, always inspect them to ensure you’re not dropping valid data. Context matters.</em></blockquote><h3>2. Anomaly Detection (Outlier Identification)</h3><p>Outliers are data points that differ significantly from the rest. They can indicate:</p><ul><li><strong>Data entry errors</strong> (e.g., a sales amount of 50,000 instead of 500).</li><li><strong>Unusual but valid events</strong> (e.g., a customer placing a bulk order).</li><li><strong>Changes in behavior</strong> worth investigating (e.g., spikes in website traffic).</li></ul><p>Detecting outliers before analysis is crucial, because leaving them unchecked can <strong>skew averages, distort trends, and mislead insights.</strong></p><h4>Common Oversights during Anomaly Detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIXuO0lIE2q9hceF_uopFg.png" /></figure><h4>Understanding Outlier Terms</h4><p><strong>i) Z-score: </strong>It is a measure of how far a value is from the average, in units of standard deviation. Simply, “How unusual is this number compared to the rest?”</p><p><strong>Example:</strong> Imagine you have a dataset of sales amounts, and most of the sales hover around <strong>100</strong>, with a typical variation (standard deviation) of <strong>20</strong>.</p><p>Now, someone makes a sale worth <strong>160</strong>. To check if this value is unusually high, you calculate its <strong>Z-score</strong>, which tells you how far it is from the average in terms of standard deviation units.</p><ul><li><strong>Mean (average)</strong> = 100</li><li><strong>Standard deviation</strong> = 20</li><li><strong>Sale amount</strong> = 160</li></ul><p><strong>Z-score formula</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*Dp2paQ0QvOnR1GRJ1eAurQ.png" /></figure><p>This means the sale is <strong>3 standard deviations above the mean</strong>.</p><p><strong>What does that mean?</strong></p><p>In a normal distribution:</p><ul><li>Most values fall within ±2 standard deviations.</li><li>Values beyond ±3 are rare and they’re considered <strong>outliers</strong>.</li></ul><p>So, a Z-score of 3 says:</p><blockquote>“This sale is way higher than usual as it’s statistically unusual and might be worth a second look.”</blockquote><p><strong>ii) Standard Deviation (STD or SD): </strong>Standard deviation (SD) measures how spread out your data is from the average (mean). It tells you whether your values are tightly clustered or widely scattered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*wwfXbHXm7etR_ZVfE0qRjA.png" /><figcaption>Standard Deviation Formula</figcaption></figure><p><strong>Explaining the above formula in plain terms:</strong></p><ol><li>Find the average of your data.</li><li>Subtract the average from each value (this shows how far each one is from the center).</li><li>Square those differences (so negatives don’t cancel out).</li><li>Add them up, divide by n−1n - 1, and take the square root.</li></ol><p>Now, imagine you run a small online store and track daily sales: 90, 100, 110. If you calculate the average of the above 3 sales, you get 100 right? and the standard deviation is 10, meaning your sales typically stay within $10 of the average, showing consistency. How did we get 10?</p><p>sales values are: 90, 100, 110</p><ul><li>Mean = 100</li><li>Differences from mean = -10, 0, +10</li><li>Squared differences = 100, 0, 100</li><li>Average of squared differences = 100</li><li>Square root of 100 = <strong>10</strong> → That’s your standard deviation.</li></ul><p>Now imagine sales of 20, 100, 180. Same average, but the standard deviation jumps to 80 (you can calculate this yourself) revealing wild swings. So while both sets average $100, one is stable, the other is unpredictable. That’s exactly what standard deviation helps you spot.</p><p><strong>iii) IQR (Interquartile Range):</strong> The range between the 25th percentile (Q1) and 75th percentile (Q3), basically the middle 50% of your data.</p><p>Simply, “Where most of my data lives; anything far outside might be unusual.”</p><p><strong>Outlier rule:</strong> Anything below Q1 - 1.5*IQR or above Q3 + 1.5*IQR is flagged.</p><p><strong>iv) Q1 &amp; Q3 (Quartiles)</strong></p><p><strong>Q1:</strong> The value below which 25% of data falls.</p><p><strong>Q3:</strong> The value below which 75% of data falls.</p><p>Together with IQR, they help you spot extreme values without assuming normal distribution.</p><h4>Example Dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*EXo_yaLhwaVssaRMuh-I2Q.png" /></figure><p>Looking at the sample sales data above, you’ll notice that <strong>CustomerID 103</strong> has a much higher sales amount, which implies an outlier. What you’ll do now is to decide whether it’s an error or confirm if it is an unusual but valid transaction before proceeding.</p><h3>Anomaly Detection using Python (pandas + scipy)</h3><pre>from scipy import stats<br>import pandas as pd<br><br>df = pd.read_csv(&quot;sales_data.csv&quot;)<br><br># Z-score method<br>z_scores = stats.zscore(df[&#39;SalesAmount&#39;])<br>outliers = df[(z_scores &gt; 3) | (z_scores &lt; -3)]<br><br>print(&quot;Outliers using Z-score:&quot;)<br>print(outliers)<br><br># IQR method<br>Q1 = df[&#39;SalesAmount&#39;].quantile(0.25)<br>Q3 = df[&#39;SalesAmount&#39;].quantile(0.75)<br><br>IQR = Q3 - Q1<br><br>outliers_iqr = df[(df[&#39;SalesAmount&#39;] &lt; Q1 - 1.5*IQR) | (df[&#39;SalesAmount&#39;] &gt; Q3 + 1.5*IQR)]<br><br>print(&quot;Outliers using IQR:&quot;)<br>print(outliers_iqr)</pre><h3>Removing Duplicates using Excel</h3><h4><strong>A. Detect Outliers in Excel Using Z-Score</strong></h4><p>1. Add a Z-score column</p><p>Suppose your sales data is in column <strong>B</strong> (SalesAmount), rows 2–101.</p><p>In a new column (say, column C), enter:</p><pre>=(B2 - AVERAGE($B$2:$B$101)) / STDEV.P($B$2:$B$101)</pre><p>where:</p><ul><li>B2 → the current value</li><li>AVERAGE($B$2:$B$101) → average of all values</li><li>STDEV.P($B$2:$B$101) → standard deviation of the <strong>entire population</strong>.</li><li>If your data is a sample, use STDEV.S instead.</li><li>$ locks the range so you can copy the formula down the column.</li></ul><p>2. Flag outliers</p><ul><li>Outliers are typically values with <strong>Z &gt; 3</strong> or <strong>Z &lt; -3</strong>.</li><li>You can add another column:</li></ul><pre>=IF(OR(C2&gt;3, C2&lt;-3), &quot;Outlier&quot;, &quot;&quot;)</pre><p>3. Optional: Conditional Formatting</p><ul><li>Select your Z-score column.</li><li>Go to <strong>Home → Conditional Formatting → New Rule → Use a formula to determine which cells to format</strong></li><li>Formula: =OR(C2&gt;3, C2&lt;-3)</li><li>Choose a fill color to highlight the outliers.</li></ul><p><strong>Notes:</strong></p><ul><li>Make sure your range in AVERAGE and STDEV matches your data exactly.</li><li>Decide whether your data is a sample or the whole population (STDEV.S vs STDEV.P).</li><li>This approach works best when your data is roughly normally distributed. If it’s heavily skewed, consider using <strong>IQR</strong> instead.</li></ul><h4><strong>B. Detecting outliers in Excel using the IQR (Interquartile Range) method</strong></h4><p><strong>Step 1: Identify Quartiles</strong></p><p>Suppose your data is in <strong>column B (B2:B101)</strong>.</p><ul><li>Q1 (25th percentile):</li></ul><pre>=QUARTILE.INC($B$2:$B$101, 1)</pre><ul><li>Q3 (75th percentile):</li></ul><pre>=QUARTILE.INC($B$2:$B$101, 3)</pre><ul><li>IQR:</li></ul><pre>=Q3 - Q1</pre><p>You can put Q1 in, say, <strong>D2</strong>, Q3 in <strong>D3</strong>, and IQR in <strong>D4</strong>.</p><p><strong>Step 2: Calculate Outlier Bounds</strong></p><ul><li><strong>Lower bound:</strong> =Q1 - 1.5*IQR</li><li><strong>Upper bound:</strong> =Q3 + 1.5*IQR</li></ul><p>Place these in separate cells (e.g., D5 = lower bound, D6 = upper bound).</p><p><strong>Step 3: Flag Outliers</strong></p><p>Add a new column, say <strong>C</strong>, with a formula like:</p><pre>=IF(OR(B2&lt;$D$5, B2&gt;$D$6), &quot;Outlier&quot;, &quot;&quot;)</pre><ul><li>B2 → the current value</li><li>$D$5 → lower bound (locked)</li><li>$D$6 → upper bound (locked)</li></ul><p>Copy this formula down the column.</p><p><strong>Optional: Conditional Formatting</strong></p><ul><li>Select your data column (<strong>B2:B101</strong>)</li><li>Go to <strong>Home → Conditional Formatting → New Rule → Use a formula to determine which cells to format</strong></li><li>Formula: =OR(B2&lt;$D$5, B2&gt;$D$6)</li><li>Choose a highlight color</li></ul><p><strong>NOTES</strong></p><ul><li>The 1.5 multiplier is standard for detecting mild outliers. Use 3 for extreme outliers.</li><li>IQR works well even for skewed data where Z-score might fail.</li></ul><h3>Removing Duplicates using SQL</h3><pre>-- Identify outliers using standard deviation<br>WITH stats AS (<br>    SELECT AVG(SalesAmount) AS mean_val, STDDEV(SalesAmount) AS std_val<br>    FROM Sales<br>)<br>SELECT *<br>FROM Sales, stats<br>WHERE SalesAmount &gt; mean_val + 3*std_val<br>   OR SalesAmount &lt; mean_val - 3*std_val;</pre><h4><strong>Important considerations</strong> when detecting outliers in SQL</h4><p><strong>1. NULL values</strong></p><p>If SalesAmount has NULLs, AVG() and STDDEV() ignore them automatically.</p><p>But be careful if you join or filter data; you may want to explicitly exclude NULLs:</p><pre>WHERE SalesAmount IS NOT NULL</pre><p><strong>2. Skewed data</strong></p><p>The standard deviation (Z-score) method assumes roughly normal distribution.</p><p>If your data is heavily skewed (e.g., lots of small sales and few huge ones), Z-score might flag many “false” outliers.</p><p>In such cases, <strong>IQR method</strong> is safer.</p><p><strong>3. Using IQR in SQL</strong></p><p>You can calculate Q1 and Q3 using PERCENTILE_CONT (PostgreSQL, SQL Server, Oracle):</p><pre>WITH quartiles AS (<br>    SELECT<br>        PERCENTILE_CONT(0.25) WITHIN GROUP (ORDER BY SalesAmount) AS Q1,<br>        PERCENTILE_CONT(0.75) WITHIN GROUP (ORDER BY SalesAmount) AS Q3<br>    FROM Sales<br>)<br>SELECT s.*, q.Q1, q.Q3<br>FROM Sales s<br>CROSS JOIN quartiles q<br>WHERE SalesAmount &lt; q.Q1 - 1.5*(q.Q3 - q.Q1)<br>   OR SalesAmount &gt; q.Q3 + 1.5*(q.Q3 - q.Q1);</pre><ul><li>This works even if the distribution is skewed.</li></ul><p><strong>4. Grouping considerations</strong></p><p>Often, you want outliers <strong>within a group</strong>, not across the whole table.</p><p>Example: detect outliers per ProductID or Region:</p><pre>SELECT ProductID, AVG(SalesAmount) AS mean_val, STDDEV(SalesAmount) AS std_val<br>FROM Sales<br>GROUP BY ProductID;</pre><ul><li>Then join back to flag outliers per product group.</li></ul><p>In summary, when detecting outliers in SQL,</p><ol><li>Check for NULLs</li><li>Consider distribution (Z-score vs IQR)</li><li>Decide the context (overall table vs groups)</li><li>Always inspect flagged rows: automated detection can sometimes misclassify valid extreme values as errors.</li></ol><h4>Outlier Detection Tips</h4><ol><li><strong>Check context</strong>: Not every outlier is an error; some reveal important insights.</li><li><strong>Clean your data first</strong>: Outliers are easier to identify when typos, spaces, and wrong formats are removed.</li><li><strong>Visualize</strong>: Boxplots or scatter plots can quickly highlight anomalies.</li></ol><h3>3. Consistency Checks</h3><p>Consistency checks ensure your data follows expected formats, ranges, and logical relationships. Inconsistent data can silently distort insights like negative quantities in sales, misspelled categories, or mismatched dates. These can all lead to flawed reports, forecasts, and decisions.</p><p><strong>Why does consistency matter? Let’s take a look at these instances and their relative impacts.</strong></p><ol><li>A product sold with Quantity = -2 could throw off revenue totals.</li><li>A Region spelled “Noth” instead of “North” may create separate categories in dashboards.</li><li>Missing values in key fields like CustomerID can break joins and aggregations.</li></ol><p><strong>Common Oversights</strong></p><ul><li>Negative or impossible values e.g., age &lt; 0, Quantity &lt; 0, age &gt; 500 (is the customer a vampire or witch?😄).</li><li>Misspelled or inconsistent categorical data (e.g., “East” vs “east”).</li><li>Missing or NULL values in critical columns.</li><li>Date or time inconsistencies (e.g., future shipping dates, wrong formats).</li><li>Misaligned relationships (e.g., ProductID not in Product Master table).</li></ul><p><strong>Example Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*S0qhtL056PSJ_faLztOotQ.png" /></figure><p>Notice the issues:</p><ul><li>Row 2: Quantity is negative.</li><li>Row 3: Region typo (“Noth” instead of “North”).</li><li>Row 4: Missing CustomerID.</li></ul><h3><strong>Checking Consistency using Python (pandas)</strong></h3><pre>import pandas as pd<br><br># Load data<br>df = pd.read_csv(&quot;orders.csv&quot;)<br><br># Check for negative values<br>neg_qty = df[df[&#39;Quantity&#39;] &lt; 0]<br>print(&quot;Negative quantities:\n&quot;, neg_qty)<br><br># Check for missing values<br>missing = df.isnull().sum()<br>print(&quot;Missing values per column:\n&quot;, missing)<br><br># Check categorical consistency<br>print(&quot;Unique regions:&quot;, df[&#39;Region&#39;].unique())<br><br># Optional: flag rows with invalid regions<br>valid_regions = [&#39;North&#39;, &#39;South&#39;, &#39;East&#39;, &#39;West&#39;]<br>invalid_regions = df[~df[&#39;Region&#39;].isin(valid_regions)]<br>print(&quot;Invalid regions:\n&quot;, invalid_regions)</pre><h3>Checking Consistency using <strong>Excel</strong></h3><ol><li><strong>Negative or invalid numbers</strong></li></ol><p>Add a helper column: =IF(B2&lt;0,&quot;Check&quot;,&quot;&quot;) (replace B2 with your Quantity column address). So if any cell in your quantity column is less than 0, it out puts “Check”, if it is not less than 0, it leaves the cell blank. So you can then filter to observe the rows <em>checked</em></p><p>Or: =COUNTIF(B:B,&quot;&lt;0&quot;) to get a quick count.</p><p><strong>2. Missing values</strong></p><p>Use =COUNTBLANK(B:B) to identify empty cells.</p><p><strong>3. Categorical consistency</strong></p><p>Use <strong>Data → Data Validation</strong> to restrict allowed values.</p><p>Use =UNIQUE(Range) (Excel 365) to inspect all distinct entries.</p><h3>Checking Consistency using <strong>SQL</strong></h3><pre>-- Flag invalid or missing data<br>SELECT *<br>FROM Orders<br>WHERE Quantity &lt; 0<br>   OR Quantity IS NULL<br>   OR Region NOT IN (&#39;North&#39;,&#39;South&#39;,&#39;East&#39;,&#39;West&#39;);</pre><p><strong>Tips:</strong></p><ol><li>Always review flagged rows before correcting because they may reveal legitimate edge cases.</li><li>Combine consistency checks with duplicate and outlier detection for cleaner datasets.</li><li>Consider building automated checks if you work with recurring reports or live dashboards.</li></ol><h3>4. Putting It All Together…</h3><p>Once you understand your dataset, it’s time to systematically check it before diving into analysis. A structured approach ensures nothing slips through the cracks:</p><p><strong>a. Load and inspect your dataset</strong></p><p>Understand the structure: columns, data types, missing values.</p><p>Look at sample rows to get a feel for the data.</p><p><strong>b. Run duplicate detection</strong></p><p>Decide whether duplicates are errors or valid repeat entries.</p><p>Remove or correct them based on your analysis goals.</p><p><strong>c. Identify anomalies</strong></p><p>Detect outliers using Z-score or IQR.</p><p>Decide: are these genuine events worth investigating, or errors to correct?</p><p><strong>d. Check for consistency</strong></p><p>Ensure numeric ranges make sense (e.g., no negative quantities).</p><p>Validate categorical values (e.g., regions, product codes).</p><p>Handle missing or NULL values appropriately.</p><p><strong>e. Document your findings</strong></p><p>Keep a clean, validated version of your dataset.</p><p>Log what you corrected, removed, or flagged for review. This makes your analysis reproducible and credible.</p><p>Skipping these checks can have serious consequences, some of which include:</p><ol><li>Misleading reports</li><li>Broken dashboards</li><li>Incorrect model predictions</li><li>Poor business decisions</li></ol><p>In real projects, even a single overlooked issue can skew the entire analysis. I’ve seen datasets where one missing or mis-entered value shifted all the results. So, catching it early saves hours of rework and maintained trust in the findings.</p><h3>After validating and cleaning your data, what is the next step?</h3><ol><li>Perform <strong>Exploratory Data Analysis (EDA)</strong> to uncover trends and patterns.</li><li>Build <strong>dashboards in Power BI or Excel </strong>for visualization and reporting.</li><li>Train <strong>predictive models in Python or Excel</strong> to drive data-backed decisions.</li></ol><blockquote>Good analysis starts with clean, reliable data<strong>.</strong></blockquote><p>Loved this? Read more on this series here: <a href="https://dominionakinrotimi.medium.com/what-they-dont-tell-you-about-real-world-data-dd0dc661ad50">What they don’t tell you about real-world data | by Dominion | Medium</a></p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7cfad148cff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How One Small Step Changes Everything]]></title>
            <link>https://dominionakinrotimi.medium.com/how-one-small-step-changes-everything-8e46a8401ceb?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e46a8401ceb</guid>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[life]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Wed, 12 Nov 2025 10:02:24 GMT</pubDate>
            <atom:updated>2025-11-12T10:02:24.658Z</atom:updated>
            <content:encoded><![CDATA[<p>Learning by Doing. Episode 5</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mq6h-oOj0znZlpAwLqe4hw.jpeg" /><figcaption>Background in the cover image by Freepik</figcaption></figure><p>Sometimes I wonder why it’s so hard to stay consistent.<br>You set a goal, even make elaborate plans, and maybe even watch a few videos about it. You tell yourself, “Okay, this time, I’ll actually do it.”</p><p>And for a while, you do. Then somehow, it just fades.</p><p>For one and a half years, I’ve been struggling to stay consistent with things like exercise, reading on subjects outside my regular themes, and studying. I know why I need to do them, I even have strong reasons, but it’s still not as easy as it sounds when people say, <em>“Just find your reason and you’ll stay consistent.”</em></p><p>Honestly, finding a reason doesn’t automatically make you act on it.<br>At least, let’s say, not all the time.</p><p>To be sincere, most people don’t say it but the truth is that those who seem “disciplined” didn’t just wake up one day and figure it out. They also struggled, maybe not in the same way, but they did. You just don’t see the parts of their process that didn’t look clean or organized.</p><p>So when people make it sound like consistency is just about motivation, it doesn’t sit right with me. It’s not, abeg.</p><p>There are days you genuinely <em>want</em> to do what you planned, but your body, your emotions, or your mind just don’t align. We’re not robots. We’re human. And humans don’t run on code, we run on patterns, moods, energy, and environment.</p><p>I’ve been learning slowly to stop expecting myself to function the same way every day. To stop setting routines that ignore how my body or mind actually works.</p><p>Even with all that, there are still small rules that help.<br>For example, I once saw this simple rule from a YouTube video: “Don’t think about it; just count 1, 2, and move before you count 3.” Like when you’re lying down and you know you should get up to maybe arrange ypur clothes that have been on that chair for weeks. Not like the chair is complaining, but because you said you’d be tidy. Don’t overthink it. Just count and start working on it before the count of 3.</p><p>It sounds simple and maybe silly, but it actually works. For instance, when I know I am to work on something and I just want to sleep on my bed and complete that movie/novel, it suddenly occurs to me that “Hey, I haven’t done this work yet.” Instead of saying “I’ll do it later” or “I’ll do it after I rest a little.” I start to count and get up before 3.</p><p>It works, but not always. I still procrastinate sometimes, and I still do last-minute work, but it has helped me move when my brain tries to talk me out of it. It still counts as something.</p><p>And that’s what I mean by being realistic.</p><p>Progress doesn’t have to look perfect. You can grow even if it’s not in an orderly way. What matters is that it adds up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eUB3kF_72y9zM3Ml.jpg" /><figcaption>Image from davidjpfisher.com, Progress Isn’t a Straight Line</figcaption></figure><p>Over the weekend, I was working on something that had a lot to do with figures. There was a specific number I was told to work with, and I knew that every single piece mattered. The data wasn’t perfect, but it was manageable.</p><p>When I finished and sent it in, the first feedback I got was that … the specific number of instances. I checked and noticed that the number of instances was off by just one. That one thing affected every other number that followed. Not in a dramatic way, but enough to make all the results shift.</p><p>And this is the same with life.<br>A single thing we overlook, something tiny, maybe even harmless, can end up shifting the bigger picture.</p><p>People talk about growth a lot. About being consistent, showing up, and staying disciplined. <br>But in real life, it’s not always that straightforward. You can write down plans, create timetables, set reminders, and still not follow through. And it’s not because you’re lazy or unserious. Sometimes, it’s because what you’re trying to be consistent with doesn’t fit into how you actually function.</p><p>You can’t force consistency with something that doesn’t align with your rhythm. So, I’ve been talking about consistency since, but how do we actually stay consistent?</p><p>I have tried a lot of things over the years, and I’ll be sharing a few practical tips that worked for me:</p><ol><li><strong>First identify the problem. </strong>Take out time to sit yourself down and draft out what is it in particular that you are struggling with. What are the things stopping me from solving it? Is it sleep? Oh, I love sleep so much. It was always on my list of hobbies in high school, and I needed to study more to improve my grades, but then I knew it sleep was hindering me from getting things done. I identified the problem. This is the first step. You can’t solve what you haven’t defined. And this also encompasses your “why”.</li><li><strong>Observe your patterns without trying to fix them.</strong><br>For one week, just watch how your day naturally flows. What time do you genuinely feel alert? What kind of work drains you? What gives you energy again? Don’t rush to fix or judge anything oh, just notice. For instance, maybe people have told you, “You take too long to finish things,” or “You always overthink before starting.” And you’ve always felt like, <em>No, I just like doing things well.</em> But you’ve heard that same feedback more than once. There’s probably something there. It doesn’t mean they’re completely right, but it also doesn’t mean they’re completely wrong. It just means there’s a pattern somewhere that you haven’t fully noticed yet. Maybe your process really does take longer, or maybe you need more time to start because you think deeply before you move. Either way, don’t fight it just understand it. Or it could even be something you have seen taking place consistently in your life, like “missing deadlines,” “low results,”… What is your life like at the moment? When you know how you actually move, it becomes easier to build systems that work for <em>you</em>, not against you.</li><li><strong>Work with your rhythm, not against it.</strong><br>This is the next step. If you’re not a morning person, stop forcing 5 a.m. routines. “Wake up by 4am today, do this, do that,” just because you saw it in a video on YouTube? If you focus best in the evening, start there. Build your structure around what already works for you.</li><li><strong>Start with one small thing you can actually keep up with.</strong><br>Don’t overthink it. Just pick one thing, one thing that won’t make you roll your eyes tomorrow. Like me, there was a time I said I’d start exercising every morning. 30 minutes. Every day. I even set alarms, wrote it on paper, and all that. It lasted three days. Then I said, &quot;Oh, it didn’t work, let me do 60 every Saturday or maybe 20 every 3 days.&quot; That one lasted for 2 months, I think. It was always one thing after the other. “My friend is around”, “I have more important things to do”, “I’m tired”. What ended up working was something small 5 -10 squats before my bath. That’s all. Some days I even forget to count, but I still do <em>something</em>. It sounds unserious, but that tiny thing made me start moving again. It’s not about discipline in the loud sense. It’s about building tiny habits, the kind that fits quietly into your day until it just becomes part of you.</li><li><strong>Track what feels natural, not what looks impressive.</strong><br>Forget perfection. Just notice what actually works for you. If something keeps failing, it’s not that you’re inconsistent oh, it might very well be your system telling you that it doesn’t fit your pattern yet. Pay attention to what actually feels natural to you.<br>Not what everyone says you <em>should</em> be doing. For me, I’ve realized there are some things I just won’t do in the morning, no matter how many routines I try to copy. I used to feel bad about it, telling myself, “You missed it again.” I tried and tried, but honestly, it’s not that I’m lazy oh, it’s just not my rhythm. I work better in the evenings. So instead of forcing myself into someone else’s “productive hours,” I just stopped fighting it.</li><li><strong>Let your growth breathe.Don’t suffocate yourself</strong><br>Some weeks will be off. Life happens, tiredness, emotions, sickness, anything. Don’t reset everything because of that. Just pick up where you are.</li></ol><p>That one missing instance from my data that I overlooked changed everything in my report.<br>This is proof that small things actually make an impact. And in the long run, every detail adds up. But you don’t fix everything at once. You fix one small thing after the other. You don’t fix everything at once.<br>I don’t always get it right. Some days, I forget. Some days, I intentionally decide to not do what I am to do for no good reason “Lemme skip a day, I’ll do it tomorrow”. Some days, I don’t move when I planned to, or I leave a task halfway, or I do it last minute.</p><p>In dev. it’s like shipping an mvp and adding updates as you go. You don’t need a perfect plan or a full routine. Just be conscious and notice the small thing that’s off, and you do that one thing. That one thing is enough to start moving the rest into place.</p><p>The more you notice the small things and act on them, even in tiny ways, the more it builds.</p><p>See, don’t expect yourself to be perfect tomorrow or the day after.</p><p>Before I end this, something just came to mind. I remember when my friend bought a new skincare product for her face, and after applying it for the second time, she came to me and said, <em>“Dominion, how is my face? It’s glowing and smooth, abi?”</em></p><p>I told her, <em>“Your face is still the same, nothing happened to it.” She was pained ehn and was like, “You can’t even encourage someone.” </em>But I told her the truth<em> </em>because that’s what I saw. Don’t lecture me on empathy here, please. 😅 I know, but realistically, this was something she had just started using, something that would take weeks to actually show its effect. She was disappointed because she expected immediate results. The product might have already started working, but I only told her what I could see.</p><p>Growth, progress, and consistency … these things don’t happen overnight. You don’t always notice the effect of the small things you do today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cBVc5nGe7EwkAJff.jpg" /><figcaption><a href="https://basonasset.com/2017/11/08/tiny-steps/">Tiny Steps — Bason Asset Management</a></figcaption></figure><p>Even when it’s invisible, even when it feels slow, I tell you, it’s working. My mentor will always say, “No effort is wasted. Every little thing matters.” And the more you act on the little things, even when it feels not on time, not on schedule… the more you act and do not abandon it because you missed something you set, the more they build quietly, eventually showing results in ways you’ll notice over time.</p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e46a8401ceb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Day I Stopped Being a Nice Leader and My Team’s Performance Skyrocketed.]]></title>
            <link>https://dominionakinrotimi.medium.com/the-day-i-stopped-being-a-nice-leader-and-my-teams-performance-skyrocketed-3f5523b61c05?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f5523b61c05</guid>
            <category><![CDATA[learning-by-doing]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[life-lessons]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 00:57:21 GMT</pubDate>
            <atom:updated>2025-10-21T00:57:21.624Z</atom:updated>
            <content:encoded><![CDATA[<p>Learning by Doing. Episode 4</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0XZv5WvWmqbV_wf26MzuHw.png" /></figure><p>I’ve had some pretty wild leadership experiences.<br>And honestly, I still feel like I have a whole lot to learn.</p><p>Leadership is not something you master in one project or one position.</p><p>It’s like a journey that keeps humbling you every single time you think, “Yeah, I’ve got this figured out.”</p><p>There was this one project I led that really tested my patience and taught me more about people, communication, and leadership than any book ever could.</p><p><strong>The Team I Didn’t Want to Lead</strong></p><p>Let me be honest, if I had the choice to pick my team members, I wouldn’t have picked that team.</p><p>It’s not because I’m looking down on them or anything.</p><p>It’s not because they were not smart.</p><p>But, when you see something that might stress you, you will know. I already knew what was coming.</p><p>I just knew; I had the feeling that they were going to stress me.</p><p>Maybe it was because I believed they were going to stress me that everything turned out the way it did sef. Who knows?</p><p>But I didn’t have a choice.</p><p>So I said, “Okay, let’s roll with it. We’ll figure it out.”</p><p>They all agreed that I should become the group lead.</p><p>The first month was fine.</p><p>We were active, communicating, working, and brainstorming. Ideas overflowing. Everything seemed cool.</p><p>But by the second month, everything just went flat. You could hear a pin drop in that group chat. It was as though I was conversing with just myself. Sending messages and responding to them yourself most times. In a team of more than 8?</p><p>You’d send a message to the group… SILENCE.<br>You’d go through the stress of tagging someone, and still silence.</p><p>Call for meeting… nothing<br>And when someone finally replied, it wouldn’t even be anything concrete. Imagine asking a question and someone reacting 👍 to your message.</p><p>So what happens? You end up doing most of the work, including making decisions yourself, because deadlines will not wait for anyone.</p><p>I explained the situation to them and even sent voice notes.<br>First time, nothing.<br>Second time, still nothing.</p><p>They were reading the messages… I could see that they were reading it. The pain that comes with the realization that they just decided not to respond. At that point, I was just tired. Tired of trying to motivate people who didn’t seem interested to me.</p><p>So one day, I opened the group chat and started removing them one by one.<br>Every. Single. One.</p><p>I left only a few who were still active.</p><h4><strong>This is the funny part</strong></h4><p>Not long after, not even up to 30 mins oh. My phone started buzzing.<br>Notifications everywhere.</p><p>In my mind I was like, “Ehn ehn? So these people can actually send messages.”<br>“Why did you remove me from the group?” “I noticed you removed me from the group. Is anything the problem?”</p><p>Yesssssss, everything is the problem sir. Very funny questions 😅</p><p>I saw the messages.</p><p>I viewed them.</p><p>But I didn’t respond. (I can be very petty, I know it.)</p><p>I know how to ghost too.</p><p>Then the apologies started rolling in:<br>“I’m sorry, I know I’ve not been active.”<br>“How can I help”<br>“What should I do”</p><p>At that point, I forgot that I was vexed. I was just laughing<br>So you knew why you were removed and wanted to prove right by asking me why I removed you from the group initially? So you knew that you weren’t being active?</p><p>Like, people know what they are doing. (In capital letter) There’s nothing you want to tell me</p><p>It was funny because the same people who ghosted the group chat suddenly had time to message me.</p><p>Then one lady reached out and said, “I’m so sorry. I’ve not been feeling fine for some time now.”<br>And I told her, “You could have told me. You could have communicated. Next time, please just let me know so I don’t take it the wrong way.”</p><p>Because, truly, I had no idea she was unwell. From my end, it just looked like she was ignoring the project. That day, I learned another thing.</p><p>Silence can be misinterpreted, and communication can save a lot of misunderstanding.</p><p>Someone even said, “He didn’t know what to say or contribute.”</p><p>And I didn’t know.</p><p>I’m not a mind reader guys 😢</p><p>After all that, they apologized and promised to do better. So I added them back to the group.</p><p>And honestly, the difference was clear. The energy changed. The teamwork improved. They started showing up.</p><p>Even the same people that were once unresponsive were now suggesting ideas, taking initiative, and actually following through. It was like a whole new team.</p><p>Then came the submission day.<br>These guys were on fire. I was shocked. Like, so you people can actually be like this? 😂<br>That was how I felt till the very end. They were the best.</p><p>They were running around, making sure everything was ready. My own laptop battery was even low, and they were literally trying to help with whatever they could. The energy was giving.</p><p>The teamwork was beautiful.</p><p>And when we finished that presentation, I just smiled quietly because “Mo ti bo” (I was free from them) and because I learned a lot.</p><p>Funny enough, the entire class already knew what had happened. Everyone was like, “Dominion removed them from the group!” 😅</p><p>It became a gist.</p><p>I was a bit embarrassed because I felt I went too extreme, but so far it worked, I was glad and relieved. It was a leadership moment that stuck.</p><p>Leadership Lessons I Took Away from That Experience</p><h4><strong>Sometimes, tough love works.</strong></h4><p>People don’t always listen to words. Sometimes they only understand action. Removing them wasn’t about punishment it was about accountability. It showed that the project mattered, and so did their contribution. I learned it from my ma. She lets us realize the consequences of actions we take with both words and actions.</p><h4>You can’t force collaboration</h4><p>There’s this … Is it a proverb? It says, “You can only force a horse to the river, but you can’t force it to drink.” You can’t force collaboration, but you can set the tone.<br>As a leader, your job isn’t to carry everyone on your head. It’s to set standards and hold the team to it. You can be kind and still be firm.</p><h4>Communication changes everything</h4><p>When that lady said she wasn’t feeling fine, I realized how easily silence can be misread. Communication isn’t just a soft skill, it’s a leadership necessity. I realized I hadn’t created a safe space for them to share their struggles early on. I was managing a group of students recently, and one thing I’d say has really helped us get far and helped me understand them better is the fact that we communicate. I told them clearly before we started … These are the rules. If anything happens and peradventure it makes you not abide or meet the standard, inform me before, during, or after so we can figure things out together. I actively checked in and made it clear that ‘I’m not feeling well’ or ‘I’m overwhelmed’ are valid reasons that won’t be judged.</p><h4>Growth often hides behind discomfort.</h4><p>I personally do not like stress. But what doesn’t stress and stretch you will not make you grow. That whole experience was stressful, but it made me grow. It made them grow too. We all walked away from that project with a deeper understanding of teamwork, responsibility, and leadership.</p><h3>The Leader I Was vs. The Leader I Became</h3><p>#Before</p><ul><li>I equated leadership with being liked and avoiding conflict</li><li>I carried the entire mental load alone, making excuses for everyone. I’m very good at this. This making up excuses for people.</li><li>I interpreted silence as personal rejection rather than a systemic issue</li><li>I waited for motivation to strike them naturally</li><li>I managed tasks, not people</li></ul><p>#After</p><ul><li>I learned that respect matters more than being liked in the moment</li><li>I distributed ownership and made accountability visible</li><li>I created systems for communication instead of hoping for it</li><li>I understood that clear consequences can be more motivating than empty encouragement</li><li>I led people, not just the project</li></ul><p>To most people, according to them, I took it too seriously, I was harsh and wicked… because why would I remove my team members from a project? My “niceness” was actually a form of cowardice. I was afraid to have difficult conversations, so I avoided them until the situation became extremely stressful for me. The day I removed them from the group was the day I stopped managing my discomfort and started leading the team.</p><p>I’m not advising you to remove your team members from the group oh. 😄 Neither am I saying I’d do it again because I have experienced it, know what I didn’t do right, and won’t make the same mistakes again. I learned the hard way that leadership isn’t just about doing the work or giving orders.</p><p>Leadership is not about being the oga (lead).</p><p>It is about managing people, emotions, accountability, and communication all at once. I’m still learning and figuring leadership out.</p><p>And trust me, it’s not for the faint-hearted.</p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f5523b61c05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t Trust Your Model’s Accuracy Score]]></title>
            <link>https://dominionakinrotimi.medium.com/dont-trust-your-model-s-accuracy-score-58575ae83ba3?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/58575ae83ba3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 23:27:23 GMT</pubDate>
            <atom:updated>2025-10-20T23:27:23.078Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Dairies. Episode 3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7gIPXVsNZEiYBrO-cxwpA.png" /></figure><p>One would think I am against your model having a high accuracy as this is the second article I am writing on this topic.</p><p>You’ve done it. You scaled the data acquisition phase, the preprocessing, transformation and finally the trainig has come to an end. Then you evaluate, the loss curve is a beautiful, smooth descent into what looks like perfection. And there it is, Validation Accuracy = 95%. Yipee!! A delightsome reward, right?</p><p>It’s a moment of pure triumph. You feel like a wizard . I most times feel like a witch. You’re ready to deploy this digital genius into the real world.</p><p>And then… it all falls apart.</p><p>I’ve had quite a number of people reach out to me recently on the subject matter “My model had a very high accuracy. It is too good to be true” and I have also experienced this several times.</p><p>The model that aced your clean, curated test set starts making baffling, embarrassing errors on real-world data. Errors that stain the white of your evaluation results. The 95% not 95ing at all. What happened?</p><p>You, my friend, have fallen into the Accuracy Trap. You trusted a single number, and that number lied to you. In this guide, we’re going to move beyond the seductive simplicity of accuracy (ouuuuu, I sound like a poet … well you’ll also a learn a few tips on how to build models you can <em>actually</em> trust.</p><p>Very well then, let’s explore some of the toolkits that reveal the truth about your model’s capabilities.</p><h3>The Seductive Lie of a Single Number #Accuracy</h3><p>Just to be clear, accuracy isn’t useless. It’s a great high-level summary. The problem it is … should I say vague? It tells you how many your model got right, but it tells you absolutely nothing about how it got them right or, more importantly, which ones it got wrong.</p><p>It is like judging a chef solely on how many dishes they can cook in an hour. It doesn’t tell you if the food is delicious, nutritious, or if they’re just serving everyone nonsense.</p><p>Getting a high accuracy is good but it costs a lot more than you can imagine in the real world. The cost of getting in wrong is extremely high in applied ML. That is, machine learning applied to various facets of life like health, finances, production, etc. and the cost of a False Positive (a false alarm) is very different from the cost of a False Negative (a missed detection). I’ll explain what a false positive or negative means better below.</p><p>Other examples include:</p><ol><li>Spam Filtering</li></ol><p>A False Positive: when a legit email is marked as spam. This is very annoying especially when you end up missing the message.</p><p>A False Negative: having spam delivered in your inbox can be annoying but is a minor inconvenience.</p><p>2. Medical Diagnosis</p><p>A False Positive: Giving a false cancer diagnosis can causes immense stress and unnecessary procedures. Imagine donating your belongings because your model said you have stage 4 cancer and will soon die (God forbid) and it now ends up as a false positive diagnosis. You’d atleast thank God for life but it won’t be funny at all.</p><p>A False Negative, that is, a missed cancer is catastrophic.</p><p>The issue now that I’m trying to emphasize is that, accuracy treats all errors equally. The real world does not. To get an understanding of our errors, we need our first diagnostic tool.</p><h3>The Confusion Matrix</h3><p>The Confusion Matrix is the antidote to accuracy’s vagueness. It’s not just a chart, it’s a storybook of your model’s performance. It breaks down predictions into four crucial categories:</p><ul><li>True Positives (TP): The model correctly predicted the positive class. (It said “fraud” and it was fraud.)</li><li>False Positives (FP): The model incorrectly predicted the positive class. (It said “fraud” but it was legit.) <em>Also known as a Type I error.</em></li><li>False Negatives (FN): The model incorrectly predicted the negative class. (It said “legit” but it was fraud.) <em>Also known as a Type II error.</em></li><li>True Negatives (TN): The model correctly predicted the negative class. (It said “legit” and it was legit.)</li></ul><p>Let’s make this concrete with a classic dataset: MNIST digits. We’ll train a simple model to answer one question: “Is this digit a zero?”</p><pre># Import libraries for data visualization<br>import matplotlib.pyplot as plt  <br>import seaborn as sns           <br># Import libraries for machine learning tasks<br>from sklearn.model_selection import train_test_split <br>from sklearn.linear_model import LogisticRegression <br>from sklearn.datasets import load_digits              <br><br># Import metric for model evaluation<br>from sklearn.metrics import confusion_matrix          <br><br># Load the digits dataset (contains images of handwritten digits 0–9)<br>digits = load_digits()<br>X, y = digits.data, digits.target <br><br># Convert the multiclass problem into binary classification<br>y_binary = (y == 0).astype(int)  # 1 if digit is zero, 0 otherwise<br><br># Split the dataset into training and testing sets (70% train, 30% test)<br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y_binary, test_size=0.3, random_state=42<br>)<br><br># Initialize and train a logistic regression model<br>model = LogisticRegression(max_iter=1000) <br>model.fit(X_train, y_train)<br><br># Predict labels for the test set<br>y_pred = model.predict(X_test)<br><br># Compute the confusion matrix to evaluate classification performance<br>cm = confusion_matrix(y_test, y_pred)<br><br># Visualize the confusion matrix using a heatmap<br>plt.figure(figsize=(5, 4))  # Set figure size<br>sns.heatmap(<br>    cm,<br>    annot=True,            # Display numbers in cells<br>    fmt=&quot;d&quot;,               # Format as integers<br>    cmap=&quot;Blues&quot;,          # Use blue color gradient<br>    xticklabels=[&quot;Not Zero&quot;, &quot;Zero&quot;],  # Predicted labels<br>    yticklabels=[&quot;Not Zero&quot;, &quot;Zero&quot;]   # Actual labels<br>)<br>plt.ylabel(&#39;Actual Label&#39;)       # Label for y-axis<br>plt.xlabel(&#39;Predicted Label&#39;)    # Label for x-axis<br>plt.title(&#39;Confusion Matrix: Is it a Zero?&#39;)  # Title of the plot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*3m8ymLh3bAv8LTCoSpiJ4g.png" /><figcaption><em>Output: A clean, annotated heatmap showing the counts of TP, FP, FN, TN.</em></figcaption></figure><p>This confusion matrix shows how well your model can tell whether a digit is a zero or not. Out of all the test examples:</p><ul><li><strong>52 times</strong>, the model correctly predicted “Zero” as “Zero” (true positives).</li><li><strong>487 times</strong>, it correctly said “this is not a zero” (true negatives).</li><li><strong>1 time</strong>, it missed a zero and said it wasn’t a zero (false negative).</li><li><strong>0 times</strong>, it mistakenly said a non-zero was a zero (false positive). That is, there was no false positive.</li></ul><h3>What does this mean?</h3><p>It means that the model is doing an excellent job. It’s very careful, it never falsely identifies a non-zero as a zero which is important if you want to avoid false alarms. However, it did miss <strong>one actual zero</strong>, which might have looked unusual or messy.</p><p>If you only looked at accuracy, you’d see a high score and assume everything’s perfect. But the confusion matrix gives you a deeper view. it shows not just how many predictions were right, but <strong>what kind of mistakes</strong> the model made.</p><p>This helps you decide what to improve. For example:</p><ul><li>If missing a zero is a big problem, you might want to boost <strong>recall</strong>.</li><li>If falsely calling something a zero is worse, you’d focus on <strong>precision</strong>.</li></ul><p>What is recall and precision? Let’s break them down.</p><h3>2. Precision, Recall, and the F1-Score</h3><p>We already understand that the “confusion matrix” gives us the raw data behind our result. Precision and Recall are the metrics that translate that data into actionable insights. They help you navigate the trade-off between different types of errors based on your project’s goals.</p><ul><li><strong>Precision</strong>:<br>Precision = TP / (TP + FP)</li></ul><p>Simply put, what precision asks is, “of all the times the model said positive, how often was it correct?”</p><p>When the cost of a False Positive is high. (e.g., recommending a risky surgery, triggering a costly fraud investigation, sending a marketing email to a disinterested customer), you optimize for precision</p><ul><li><strong>Recall</strong>:<br>Recall = TP / (TP + FN)</li></ul><p>While for recall, it simply asks: “Of all the actual positive cases, how many did the model successfully find?”</p><p>When to optimize for it: When the cost of a False Negative is high. (e.g., cancer screening, failing to detect a security breach, missing a defective product on an assembly line).</p><ul><li><strong>F1-Score</strong>:<br>F1 = 2 * (Precision * Recall) / (Precision + Recall)</li></ul><p>F1-Score is a number between 0 and 1 that tells you how well your model is performing especially when you’re dealing with imbalanced data (where one class appears much more than the other). It combines two important metrics:</p><ul><li><strong>Precision</strong>: How many of the predicted positives were actually correct?</li><li><strong>Recall</strong>: How many of the actual positives did the model successfully find?</li></ul><p>F1-score is the harmonic mean of these two. That means it gives a balanced score that’s only high when both precision and recall are high.</p><p>Let’s Calculate Them:</p><pre>from sklearn.metrics import classification_report<br><br># Print a full report directly from our predictions<br>print(classification_report(y_test, y_pred, target_names=[&quot;Not Zero&quot;, &quot;Zero&quot;]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*4Zu9iRkxUMVHY9TUsoO2wg.png" /><figcaption>Output: A clean table showing precision, recall, and f1-score for each class, along with support (number of instances).</figcaption></figure><p>This report shows that the model is performing exceptionally well and it also tells you exactly where your model’s strengths and weaknesses lie. Now you can make an informed decision. Note this is a sample not a benchmark oh.</p><p>Let’s explore more advanced diagnostic techniques</p><h3>The Black Box Problem. Is Your Model Right for the Right Reasons?</h3><p>Another thing you need to know is that your model could have perfect precision and recall but still be a complete fraud. Just stick with me, don’t be tired.</p><p>How? It might be learning a “shortcut” or a “spurious correlation” in your data instead of the actual concept you want it to learn.</p><h3>What Is Shortcut Learning?</h3><p>Shortcut learning happens when a model finds an easy pattern in the data that helps it get the right answer but not for the right reason.</p><p>Instead of truly understanding the concept, it latches onto a spurious correlation a pattern that works in training but fails in the real world. (Spurious correlation is just a fancy way of saying “two things seem related but they are not”)</p><p>For instance, a model trained to identify cows might learn to detect <em>grass</em> because most cow pictures are taken in fields. Show it a cow on a beach, and it fails. Also, a model for diagnosing pneumonia from X-rays might learn to recognize the specific metal nibs on the chest X-ray tags used at a particular hospital, not the actual medical pathology.</p><p>This is where model interpretability techniques come in. They are the “eye-tracker” for your AI, showing you what it’s actually looking at.</p><h3>Grad-CAM</h3><p>Gradient-weighted Class Activation Mapping (Grad-CAM) is a visualization technique used in Convolutional Neural Networks (CNNs). It creates a heatmap that highlights the regions of an image that were most important for the model’s prediction. If the heatmap highlights the object to be predicted, your model is learning the right thing but if it highlights backgrounds, watermarks, or irrelevant areas, your model might be using shortcuts.</p><p>To give a more specific example on how GradCam can help us understand our model better, for a model correctly identifying a cat, the heatmap should light up on the cat’s face and body right? Now, if it instead highlights a distinctive background fence that happened to be in all your training photos of cats, you’ve found a shortcut.</p><h3>Stress Testing</h3><p>As the name implies, it is stressing your model before it stresses you. A model that performs well on a clean, static test set might be brittle. To stress test a model, you deliberately expose it to challenging, noisy, or distorted inputs that mimic real-world messiness. The goal is to see how well the model holds up when things aren’t perfect because in production, they rarely are.</p><p>Here’s how it works in practice: You take your clean test set and introduce controlled disruptions.</p><p>For example, in a digit classification task, you might add random noise to the pixel values, blur the images slightly, or distort the shapes. Then you run predictions on both the original and the stressed versions.</p><p>If the model’s predictions change drastically, it’s a sign that the model is relying on fragile patterns that don’t generalize well. But if it stays consistent, you’ve got a robust model that’s more likely to perform reliably in the the real world.</p><h3>The Occlusion Test: What’s the Key Feature?</h3><p>What happens if you block the most important part of the image? This test helps you verify if the model is using the features you think it is. Occlusion testing is about hiding parts of the input like blacking out a section of an image or masking a feature in a dataset to check what the model truly depends on. If the prediction changes drastically when you remove something irrelevant, your model might be relying on the wrong clues.</p><p>Image Example: If you black out the wheels of a truck and your ‘vehicle classifier’ still confidently calls it a truck, it might be relying on the logo or background instead of the vehicle’s defining features.</p><p>Non-Image Example: In a loan application model, if removing the credit_score feature causes predictions to flip wildly, it reveals the model was using credit score as a crutch rather than learning a robust pattern from multiple financial indicators.</p><pre># Create a corrupted test set by removing one feature&#39;s information<br>X_test_corrupted = X_test.copy()<br>X_test_corrupted[&#39;feature_to_occlude&#39;] = X_test[&#39;feature_to_occlude&#39;].mean()  # Replace with average value to neutralize its influence<br><br># Generate predictions using the original and corrupted test sets<br>baseline_predictions = model.predict(X_test)               # Predictions with full feature information<br>corrupted_predictions = model.predict(X_test_corrupted)    # Predictions after occluding one feature<br><br># Measure how many predictions changed due to the occlusion<br>flipped_predictions = (baseline_predictions != corrupted_predictions).sum()  # Count mismatches<br>print(f&quot;Predictions that flipped: {flipped_predictions}/{len(X_test)}&quot;)      # Display the impact of occlusion</pre><p>If many predictions flip when you occlude a single feature, your model may be taking dangerous shortcuts rather than learning the underlying concept.</p><h3>Now, we have discussed some but we cannot discuss all at length…</h3><p>Here is a full list of techniques you can use to interrogate your model like a seasoned detective.</p><p><strong>Interpretability Techniques</strong></p><ol><li><strong>Confusion Matrix: </strong>Breaks down prediction outcomes into TP, FP, FN, TN for deeper error analysis.</li><li><strong>Classification Report: </strong>Summarizes precision, recall, F1-score, and support per class.</li><li><strong>Feature Importance Plots: </strong>Shows which features contribute most to predictions (especially in tree-based models).</li><li><strong>Grad-CAM: </strong>Visualizes what parts of an image influence CNN predictions.</li><li><strong>LIME (Local Interpretable Model-Agnostic Explanations): </strong>Explains individual predictions by approximating the model locally.</li><li><strong>SHAP (SHapley Additive exPlanations): </strong>Uses game theory to assign feature importance values.</li><li><strong>Integrated Gradients: </strong>Measures feature importance by averaging gradients from a baseline to the input.</li><li><strong>Score-CAM / XGrad-CAM / Ablation-CAM</strong> : Variants of Grad-CAM with improved reliability.</li><li><strong>Partial Dependence Plots (PDPs): </strong>Visualizes how a feature affects predictions across its range.</li></ol><h4>Robustness &amp; Reliability Tests</h4><ol><li><strong>Stress Testing: </strong>Evaluates model performance under extreme or noisy conditions.</li><li><strong>Corruption Testing</strong>: Tests model with blurred, noisy, or low-contrast inputs.</li><li><strong>Occlusion Testing: </strong>Blocks parts of the input to see if predictions change.</li><li><strong>Out-of-Distribution Testing: </strong>Checks how the model handles unfamiliar or unexpected inputs.</li><li><strong>Adversarial Testing</strong>: Introduces subtle perturbations to fool the model.</li><li><strong>Confidence Calibration: </strong>Measures how well predicted probabilities match actual correctness.</li><li><strong>Counterfactual Testing: </strong>Alters inputs slightly to see if predictions change logically.</li><li><strong>Data Shuffling or Permutation Tests: </strong>Randomizes features to detect reliance on spurious patterns.</li></ol><p>True confidence in machine learning doesn’t come from a single number. It comes from a process of rigorous interrogation. You need to question your model, poke it, prod it, and see how it reacts. It is only then that you can say you truly trust the model to perform in the wild wide world.</p><p>So the next time you see that 90 something percent validation accuracy, don’t celebrate just yet.</p><p>Don’t see it as the finish line, because <em>k’ama pa iro</em> (truth be told) it is just the beginning. Your future self, dealing with a production system, will thank you.</p><p>Now, I want you to share any of these in the comment:</p><ol><li>What’s the most surprising or funny “shortcut” your model has learned?</li><li>What you learned from this.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58575ae83ba3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Tamper-Proof Attendance System in Google Sheets (With Formulas That Outsmart Students)]]></title>
            <link>https://dominionakinrotimi.medium.com/building-a-tamper-proof-attendance-system-in-google-sheets-with-formulas-that-outsmart-students-c4a20eb74a65?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4a20eb74a65</guid>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 13:41:31 GMT</pubDate>
            <atom:updated>2025-10-01T14:46:52.571Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Dairies. Episode 2</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMQOcdhrAPURMqmpslyyZQ.jpeg" /></figure><p>Most times when it comes to recording attendance, our best bet is to set up a Google Form for daily attendance. Seems simple and straightforward, right?</p><p>Students fill it out, timestamp gets recorded, you calculate attendance.</p><p>But the problem lies in the fact that,</p><ul><li>Some students backdate entries (“I swear I was present yesterday!”).<br>Some even go ahead to take attendance for the next three days in one day. Criminals! 😅</li><li>Others submit multiple times per day (hoping extra clicks = extra marks). Sodiq geng</li></ul><p>And then your attendance % calculations get inflated.</p><p>Did it a few times as a student (I think I got away with it 😫). Been there as a tutor. Seen that. Fixed it.</p><p>In this article, I’ll show you how to build a smart, formula-driven attendance system that automatically flags:</p><p>✅ <strong>Valid:</strong>First correct submission of the day</p><p>❌ <strong>Invalid</strong> :Wrong date compared to timestamp</p><p>⚠️ <strong>Duplicate: </strong>Extra submissions that day</p><p>By the end, you’ll have a fraud-proof attendance tracker that updates automatically.</p><p>Here’s a quick workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*23dAcKqUepeR8QVxK8d79Q.png" /><figcaption>Flowchart of the tamper-proof attendance system logic. The chart visualizes the automated process for validating submissions and calculating final attendance status, ensuring data integrity against backdating and duplicate entries. (Created with Mermaid.js)</figcaption></figure><h3>Step 1: Collect the Right Data</h3><p>When you set up your <strong>Google Form</strong>, make sure it collects three things:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*gDfCgxOxe21GOi1ozAEMrw.png" /><figcaption>Image of my google form</figcaption></figure><ol><li><strong>Timestamp: </strong>This is auto-generated by Google (can’t be faked).</li><li><strong>Student ID: </strong>Unique code, e.g., TSU/DA/25/001.</li><li><strong>Entered Date: </strong>The date students claim they’re present.</li></ol><p>Your raw sheet looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*7ATKNodkgZB9VQMsetVm6g.png" /></figure><h3>Step 2: Verify the Date (Prevent Backdating)</h3><p>We first check if the student’s entered date matches the actual timestamp.</p><p><strong>Formula in D2 (Status column):</strong></p><pre>=IF(TEXT(A2,&quot;mm/dd/yyyy&quot;)&lt;&gt;TEXT(C2,&quot;mm/dd/yyyy&quot;),&quot;Invalid&quot;,&quot;Valid&quot;)</pre><p>If dates don’t match → <strong>Invalid</strong>.</p><p>If they match → <strong>Valid</strong> (for now).</p><h3>Step 3: Catch the Duplicates</h3><p>Here’s the trick: a student might submit <strong>twice in the same day</strong>. Without extra checks, both will appear “Valid.”</p><p>To fix this, we upgrade the formula:</p><pre>=IF(INT(A2)&lt;&gt;INT(C2),&quot;Invalid&quot;,<br> IF(COUNTIFS(B$2:B2,B2,A$2:A2,&quot;&gt;=&quot;&amp;INT(A2),A$2:A2,&quot;&lt;&quot;&amp;INT(A2)+1)&gt;1,&quot;Duplicate&quot;,&quot;Valid&quot;))<br></pre><h3>Breakdown:</h3><h4>IF(INT(A2)&lt;&gt;INT(C2),&quot;Invalid&quot;, ...)</h4><ul><li>A2: Timestamp of when the form was submitted (includes date and time).</li><li>C2: The date the student claims they attended.</li><li>INT(A2)<strong> and </strong>INT(C2): Removes the time portion, leaving only the date.</li><li>&lt;&gt;: Means “not equal.”</li></ul><p>If the actual submission date ≠ claimed attendance date, mark as &quot;Invalid&quot; (likely backdated or future-dated).</p><h4>COUNTIFS(...)&gt;1</h4><p>This part checks for <strong>duplicate submissions</strong> on the same day by the same student.</p><pre>COUNTIFS(<br>  B$2:B2, B2,                        → Match Student ID<br>  A$2:A2, &quot;&gt;=&quot;&amp;INT(A2),              → Match Timestamp on or after the same day<br>  A$2:A2, &quot;&lt;&quot;&amp;INT(A2)+1              → Match Timestamp before the next day<br>)</pre><ul><li>B$2:B2, B2: Looks for the same Student ID from the top of the sheet down to the current row.</li><li>A$2:A2<strong> with </strong>&quot;&gt;=&quot;&amp;INT(A2)<strong> and </strong>&quot;&lt;&quot;&amp;INT(A2)+1: Filters timestamps that fall within the same calendar day.</li></ul><p>If the student has already submitted once today, this will count more than 1 and mark as &quot;Duplicate&quot;.</p><h4>&quot;Valid&quot;</h4><p>If the dates match and it’s the first submission of the day → mark as &quot;Valid&quot;.</p><p><strong>Simply put:</strong></p><p>ConditionResultSubmission date ≠ claimed date &quot;Invalid&quot;</p><p>Same student submitted more than once today &quot;Duplicate&quot;</p><p>First correct submission of the day&quot;Valid&quot;</p><h3>Step 4: Example in Action</h3><p>After applying the formula, your sheet auto-categorizes each row:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*ez_ApGvKpnokChgP3nDqsQ.png" /></figure><p>Now it’s impossible for a student to “farm” extra attendance by clicking twice.</p><h3>Step 5: Build the Summary Report</h3><p>A summary sheet helps you track each student’s real attendance.</p><p><strong>Valid Count per student:</strong></p><pre>=COUNTIFS(&#39;Form Responses&#39;!B:B,A2,&#39;Form Responses&#39;!D:D,&quot;Valid&quot;)</pre><p><strong>Attendance %:</strong></p><pre>=ROUND((Valid_Count/Total_Days)*100,2)</pre><p>Example:</p><p>For this program, the duration is a total of 24 days.</p><ul><li>24 total program days → 1 day = 4.17%</li><li>19 days attended = 79.17%</li></ul><h3>Step 6: Pass/Fail Rule</h3><p>If your certification requires <strong>80% attendance</strong>, add this formula:</p><pre>=IF(Attendance_Percent&gt;=80,&quot;Pass&quot;,&quot;Fail&quot;)</pre><p>No arguments. No drama. Numbers don’t lie.</p><h3>Step 7: Why Does This Work and Scale</h3><ol><li>It is automatic. Every new form entry updates instantly.</li><li>It is tamper-proof. Students can’t backdate or double-dip.</li><li>It is transparent. Admins see full logs if disputes arise.</li><li>Last but not the least, it is scalable. It works for 20 or 2,000 students.</li></ol><p>With just a few smart formulas, you would have transformed a basic Google Form into a certification-ready attendance system:</p><p>Invalid entries flagged.</p><p>Duplicates neutralized.</p><p>Percentages are auto-calculated.</p><p>Your students focus on learning. You focus on teaching. And the sheet does the policing. Say cheese! ✌️ 😊</p><p>I’ll be sharing this as a <strong>free, ready-to-use template</strong> on my <a href="https://selar.com/m/dominion-akinrotimi">Selar page</a>. Keep an eye out there (and <a href="https://selar.com/showlove/ado">show some love </a>🙈) because I’ll soon be updating my services and products and yes, this attendance system template will be one of the freebies. 💃</p><p>All you’ll need to do is connect it to your Google Form responses, and you’re good to go.</p><p>Want to read more stuff like this that blends real talk + real tips?<br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4a20eb74a65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Model Isn’t Smart, It Cheated: 7 Ways Data Leakage Makes Your Test Scores Look Too Good]]></title>
            <link>https://dominionakinrotimi.medium.com/your-model-isnt-smart-it-cheated-7-ways-data-leakage-makes-your-test-scores-look-too-good-a7ff94ef7c79?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7ff94ef7c79</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 14:44:23 GMT</pubDate>
            <atom:updated>2025-07-27T14:44:23.034Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Data Diaries. Episode 1</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsb3W36HDCTOqKUKQFAvkA.jpeg" /></figure><p>Last month, I worked on training a hybrid deep learning model. The accuracy was too good to be true, the least was 98%. If it were before, i’d be super proud of myself … But something felt super off. “I must have missed something somewhere.” That was my thought. It was too good to be true, kind of. 😄 And I was right, something was off.</p><p>We’ve all been there. You train a model, run the test, and boom, 98% accuracy. Your head swells.<br>You’re about to tweet “Model successfully trained” and maybe even start writing a Medium post. But slow down. Uncle, Sister, put your hand on your chest and press it down. Repeat after me, “Calm down.” VERY GOOD!</p><p>What if I told you that score might be fake? Not because your model didn’t learn but because it learned <em>too much.</em> The wrong things. Things it shouldn’t have known.</p><p>We’ll be discussing <strong>data leakage, </strong>the silent killer of model credibility.</p><p>What is Data Leakage?</p><p><a href="https://www.bing.com/ck/a?!&amp;&amp;p=a6656a240ac56b631a5a2e0a179e5d5ed82b4b99a9b803043f05cec28a9f65adJmltdHM9MTc1MzQ4ODAwMA&amp;ptn=3&amp;ver=2&amp;hsh=4&amp;fclid=091b764a-52a4-6fad-1d49-6041531f6e97&amp;psq=Data+leakage&amp;u=a1aHR0cHM6Ly93d3cuZ2Vla3Nmb3JnZWVrcy5vcmcvbWFjaGluZS1sZWFybmluZy93aGF0LWlzLWRhdGEtbGVha2FnZS8&amp;ntb=1">In <strong>machine learning</strong>, it happens when a model learns from data it shouldn’t have access to during training, leading to misleadingly high performance during testing but poor real-world application.</a></p><p>In simple terms? It’s like giving someone the exam questions ahead of time, then acting surprised they passed. Your model isn’t smart. It just cheated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-SjP4JgdaUIuEsElLAN0Q.jpeg" /></figure><p>Let’s look at 7 common leaks.</p><h3>1. Train-Test Overlap (a.k.a. “Copy My Homework”)</h3><p>Imagine prepping a student with a set of past questions, and then during the exam, you hand them those exact same questions. Some of y’all lecturer do this. 👀<br>They’ll definitely score high… But did they actually learn anything new?</p><p>So when you train your model on all the data, including the part you were supposed to test it on. That’s not evaluation, that’s leakage.</p><p><strong>Leak Example:</strong></p><pre>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)<br>model.fit(X, y)  # You trained on everything, not just X_train</pre><p><strong>Correction:</strong><br> Always split your data <strong>before</strong> training. This is very very important.<br>Your model should only “see” the training data during .fit()</p><pre>model.fit(X_train, y_train)  # ✅ Train only on the training set</pre><p>Also, it’s better to split your data before doing preprocessing and normalization, not after. Why? Continue reading to know why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gJTqmog74YZPdNMnLCrjrA.jpeg" /></figure><h3>2. Temporal Leakage (Future Info in the Past)</h3><h4>What Is Temporal Leakage?</h4><p>It happens when your model sees information from the future, even though it’s supposed to make predictions based only on what’s known <em>now</em>. For instance, you’re building a model to predict if a customer will miss a payment next month. But one of your input features is “number of missed payments.”</p><p>That value is only available after the customer has already defaulted, meaning you’re feeding the model post-event data to make a pre-event prediction.</p><p>That’s cheating, like giving someone the answer sheet <em>before</em> the test and pretending they solved it on their own. At this point, I’d say anything leakage can be tagged as cheating. 🫣</p><p>When dealing with time-based predictions (like defaults, churn, fraud, etc.), you should:</p><ul><li>Simulate real-time conditions: only use features you would genuinely have at the time of prediction.</li><li>Avoid any fields that can only be filled in after the event you’re trying to predict.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*atD0Yjb5rOWbXacMLh4rbQ.jpeg" /></figure><h3>3. Target Leakage (Labels Hiding in Disguise)</h3><p><strong>Example:</strong> Predicting diabetes using a feature called blood_sugar_level. If blood_sugar_level &gt; 180 already defines diabetes, your model’s not predicting it’s shortcutting.</p><p><strong>Rule of thumb:</strong><br>If a feature is <strong>highly correlated</strong> with the label and wouldn’t be known before the outcome, raise eyebrows. Drop or modify the feature, or try training without it to see if accuracy drops massively. If it does, your model might’ve been using a shortcut instead of truly learning.</p><p><strong>Code check:</strong></p><pre>df.corr()[&#39;target&#39;].sort_values(ascending=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9deAUO-QaNhmESc_ruGqUQ.jpeg" /></figure><h3>4. Data Transformation Leakage (Scaling Test Data Early)</h3><p>It happens when you apply scaling or preprocessing to the entire dataset before splitting into train and test sets. That means your model gets access to statistics (like mean or standard deviation) from the test set which it shouldn’t know during training.</p><p>It’s better to split your data before doing preprocessing and normalization. That way, your model learns from just the training data and can handle new data more accurately. If you normalize before splitting, you might accidentally include hints from the test set, which can make your model look better than it really is.</p><p>Avoiding Data Transformation Leakages:</p><ul><li>Prevents inflated accuracy scores during testing</li><li>Ensures real-world reliability when deployed</li><li>Keeps your model honest, no cheating with future data</li></ul><p><strong>Leak example:</strong></p><pre># This causes leakage<br>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X)  # Uses all data, including test set<br>X_train, X_test = train_test_split(X_scaled, test_size=0.2)</pre><p><strong>Fix:</strong></p><pre># Safe and clean<br>X_train, X_test = train_test_split(X, test_size=0.2)<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)  # Learn from training only<br>X_test_scaled = scaler.transform(X_test)        # Apply same transform to test</pre><h3>5. Entity Leakage (Same Person, Both Sides)</h3><p>This happens when the same entity (like a person, user, or organization) appears in both the training and test sets. If your model sees someone during training, it might memorize their behavior so when they show up in the test set, the model isn’t generalizing, it’s just recalling.</p><p>Say you’re building a model to detect fraudulent transactions. If <strong>user ID 123</strong> appears in both training and test sets and they committed fraud in training, the model will easily flag them in testing. But that’s not learning patterns of fraud… that’s just memorizing a name.</p><p>Why It’s Dangerous</p><ul><li>Inflates performance metric, your model looks smarter than it is</li><li>Fails to generalize to <strong>new, unseen users</strong></li><li>Breaks the illusion of fairness and robustness</li></ul><p><strong>How do we fix this?</strong><br>Group-level split/ GroupShuffleSplit: This ensures that each entity only appears in one set</p><pre>from sklearn.model_selection import GroupShuffleSplit<br>splitter = GroupShuffleSplit(test_size=0.2)<br>for train_idx, test_idx in splitter.split(X, groups=df[&#39;user_id&#39;]):<br>    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]</pre><p>If your dataset doesn’t have a clear entity ID, try clustering or deduplication to group similar records. And always ask, “Would this feature exist before the prediction target?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6P7ENcuoCYqzjPfXv-Hhmw.jpeg" /></figure><h3>6. Pipeline Leakage (Mixing Prep Steps Too Early)</h3><p>This occurs when data preprocessing steps (like imputation, scaling, and encoding) are applied to the entire dataset before splitting into train and test sets. That means your pipeline learns from the test set, which it shouldn’t even know exists yet! This is similar to transformation leakage</p><p><strong>Example of pipeline leakage:</strong> imputing missing values using the <strong>mean of the entire dataset</strong>, including test set.</p><p><strong>Code Example of Leakage:</strong></p><pre># Here, the imputer calculates the mean from the full dataset including test data.<br>imputer = SimpleImputer(strategy=&#39;mean&#39;)<br>X_imputed = imputer.fit_transform(X)  # Uses all data, including test set<br>X_train, X_test = train_test_split(X_imputed, test_size=0.2)</pre><p><strong>Fix:</strong> Use pipelines</p><pre>from sklearn.pipeline import Pipeline<br>from sklearn.impute import SimpleImputer<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.linear_model import LogisticRegression<br><br>pipe = Pipeline([<br>    (&#39;imputer&#39;, SimpleImputer(strategy=&#39;mean&#39;)),<br>    (&#39;scaler&#39;, StandardScaler()),<br>    (&#39;classifier&#39;, LogisticRegression())<br>])<br><br>pipe.fit(X_train, y_train)  # All steps learn only from training data<br>pipe.predict(X_test)       # Safely applies learned steps to test data</pre><p>Everything (i.e. transformation) happens only on the training set and is learned only from the training set, then is reused on test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqZWBfZouFL_DpjISD9XQg.jpeg" /></figure><h3>7. Label Leakage via Feature Engineering</h3><p>When you <strong>create features that are derived from or highly correlated with the target label</strong>, often unintentionally, we risk the chances of the labels leaking. I mentioned above that if a feature is <strong>highly correlated</strong> with the label and wouldn’t be known before the outcome, raise eyebrows, not one. Raise both brows. 😅<br>These features give away the outcome your model is supposed to predict, turning prediction into a glorified recall.</p><p>Let’s say you are predicting loan approval with a feature like is_approved_last_year. That’s basically the answer you’re trying to predict, just shifted. It is screaming, Leakage!!!!</p><p>If a featuref explains &gt;95% of the variance in y, and f is derived from y, you’re cheating. Both you and your model are co-cheaters. 👀</p><pre>from sklearn.metrics import r2_score<br>r2_score(y, df[&#39;leaky_feature&#39;])  # Suspicious if &gt;0.9</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7o5pxCHHoKZSGOJA4WcKQ.jpeg" /></figure><p>Before you train,</p><ul><li>Are your train/test splits realistic?</li><li>Could this feature exist before the event you’re predicting?</li><li>Are there any lookaheads or time-traveling features?</li><li>Did your pipeline touch test data during prep?</li></ul><p>If you’re still unsure, try this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C82XSTTlyCuW2NN8n0P9cg.jpeg" /></figure><blockquote><em>Train your model. Then remove a “suspicious” feature and retrain. If performance drops drastically, you might’ve been depending on a leaky shortcut.</em></blockquote><h3>Final Thought:</h3><p>Great models don’t just perform well they perform honestly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Ch63--2XduhFaXO4BeuiQ.jpeg" /></figure><p>Data leakage won’t just inflate your metrics it’ll embarrass you when your model hits the real world and flops. So guard your test set. Think like a data detective. And remember:</p><p>A smart model isn’t the one that gets high scores. It’s the one that earns them.</p><p>Want to read more stuff like this that blends real talk + real tips? <br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7ff94ef7c79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Five Real-World Skills I Had to Teach Myself And Why the University Isn’t a Scam, Just Not the…]]></title>
            <link>https://dominionakinrotimi.medium.com/the-five-real-world-skills-i-had-to-teach-myself-and-why-the-university-isnt-a-scam-just-not-the-d9b25526b200?source=rss-a035a7a35e11------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9b25526b200</guid>
            <dc:creator><![CDATA[Dominion]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 14:41:55 GMT</pubDate>
            <atom:updated>2025-07-27T14:41:55.640Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Five Real-World Skills I Had to Teach Myself And Why the University Isn’t a Scam, Just Not the Whole Picture</h3><p>Learning by Doing. Episode 3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qDkDRi307X8A_u1hnFTTNg.jpeg" /></figure><p>There’s a point where “first class” becomes your entire personality. “Igi iwe”, “bookworm”, “you dey follow first class student argue?”, …</p><p>And it’s not even your fault. From 100 level, everything starts sounding like, <em>“If you finish with a good grade, the world will bend for you.”</em></p><p>Spoiler alert: it doesn’t. It is not by CGPA alone.</p><p>I’ve sat in rooms where someone with half my CGPA got the role, led the team, or closed the opportunity. And I’m not talking down on grades …no, this isn’t that <em>“school na scam”</em> conversation. I won’t lie, I love school.</p><p>But if we’re being serious, school alone is not enough.</p><p>There’s a gap. A very real one. And if you don’t catch it early, the real world will school you hard.</p><p>Here are five things I had to teach myself <em>outside</em> the classroom to stay relevant:</p><h3>1. How to Communicate (So People Actually Listen)</h3><p>Most students battle with stage fright, lack public speaking skills and communicate terribly. You might think you’re a good communicator because you present in class. But try explaining what you do to a non-tech person, a hiring manager, or even your parents and watch how fast you start stuttering.</p><p>I had to learn how to <em>translate</em> my skills. Not dumb them down just simplify without stripping the value. Your CGPA won’t speak for you in a networking event. Your words will. Your CGPA won’t communicate the value you have to offer. Your words will. A phrase I heard somewhere that I love quoting so much: “Even if you’re 70% qualified for an opportunity, if you can only communicate 30% of it, you’ll be perceived as just 30% ready.”</p><h3>2. How to Work with People (Who Aren’t Like You)</h3><p>In school, … Jesus Jesus! I’ve led groups on projects, and they frustrated heaven out of my life. I learnt patience, how to communicate, how to make demands, how to delegate, disciple, when to take action, how to handle criticism (“you are doing too much”).</p><p>As a student, take group projects seriously. Be actively involved.</p><p>In school, you pick your group members. In real life, you don’t.</p><p>I’ve worked with people who missed deadlines, disappeared during sprints (ghosters), or didn’t know half the tools I did. But guess what? We still had to deliver. I had to learn <em>collaboration,</em> not just competence. Ego doesn’t help you ship products.</p><h3>3. How to Solve Problems Without a Textbook</h3><p>In uni, if you study hard enough, you pass.</p><p>In real life? Problems don’t have marking schemes. You Google. You test. You ask stupid questions. You fail fast and adjust faster. I had to teach myself <em>resilience,</em> not just intelligence.</p><h3>4. How to Market Myself Without Feeling Like a Fraud</h3><p>Nobody tells you this, but being skilled isn’t enough.</p><p>If you can’t put it out there if your bio is dry, your LinkedIn is dead, and you only post screenshots on WhatsApp you’ll be invisible. I had to learn how to <em>show up,</em> not just <em>level up.</em> There’s a reason some people get opportunities you never hear about. Network with people. Let people know you for what you do.</p><p>Don’t be ashamed to market your services.</p><p>Bishop David Oyedepo will say “It is <strong>foolishness to be ashamed of what is gainful”.</strong></p><h3>5. How to Take Initiative Without Waiting for Permission</h3><p>School trains you to wait for lectures, for assignments, for results. But the world rewards people who move <em>first.</em></p><p>I had to learn how to start things personal projects, side hustles, community roles and follow through, even when nobody asked me to. That’s where real growth happened. That’s what made my transcript <em>useful,</em> not just beautiful.</p><h3>So… Is School a Scam?</h3><p>Not quite. University gave me structure, exposed me to my field, expanded my network, brought a lot of opportunities my way, challenged my thinking...School was literally my major source of income from year 1–3. (I’m not referring to money from my guardians.)</p><p>But it’s not the whole story. It’s a chapter.</p><p>The real issue? We treat university like the whole book.</p><p>So here’s what I think: use uni, but don’t stop there. Build in parallel. While you’re chasing the grades, also chase growth. Teach yourself soft skills, real-world context, and how to navigate beyond the four walls.</p><p>Because out here? CGPA is the bonus. Your skills, the value you have to offer, your story, and your soft power…that’s the real currency. It is literally what you have to bring to the table, and it is what can make you the owner of the table.</p><p>Want to read more stuff like this that blends real talk + real tips? <br>Do well to follow this account.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9b25526b200" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>