<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nidhin on Medium]]></title>
        <description><![CDATA[Stories by Nidhin on Medium]]></description>
        <link>https://medium.com/@nidhin.dwh?source=rss-46ed6bbe72cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ihl_HxabhrluK5hUjfwLCQ.jpeg</url>
            <title>Stories by Nidhin on Medium</title>
            <link>https://medium.com/@nidhin.dwh?source=rss-46ed6bbe72cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:01:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nidhin.dwh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[10 Machine Learning Interview Questions on Model Evaluation That Separate Juniors From Seniors]]></title>
            <link>https://medium.com/the-learning-rate/10-machine-learning-interview-questions-on-model-evaluation-that-separate-juniors-from-seniors-07e6806c9684?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/07e6806c9684</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 22:26:27 GMT</pubDate>
            <atom:updated>2026-11-15T22:26:27.537Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Anyone can print an accuracy score. These questions test whether you know when it’s lying.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gyc-7XpUnT-mjOnTRv0I4Q.png" /></figure><p>Model evaluation questions look easy. Everyone can recite a formula for RMSE or define a false positive. That’s exactly why interviewers love them: the definition is where the junior answer stops, and the senior answer only starts.</p><p>The difference is judgement. A senior candidate connects every metric to a decision: what will be done with the predictions, which mistakes are expensive, and what the number is hiding.</p><p>Here are ten questions on evaluating regression, classification and ranking models. For each one: what’s really being tested, a strong answer, and the red flag that gives a candidate away.</p><h3>1. Your regression model has a low MAE, but its mean error is +0.2. Is that a problem?</h3><p><strong>What they’re testing:</strong> whether you look at the direction of errors, not just their size.</p><p><strong>Strong answer:</strong> a residual is actual minus predicted, so a positive mean error means the model under-predicts on average. If the errors are scattered on both sides, a small positive ME may be noise. If nearly every error is positive, it’s a systematic bias: maybe a missing feature, a shift in the data since training, or a target that has grown over time. A constant bias is the easiest problem in ML to fix. Correct it on validation data, then confirm on the test set. I’d also plot residuals against the prediction and against time to see whether the bias is constant or growing.</p><p><strong>Red flag:</strong> “MAE is low, so the model is fine.” MAE can’t tell you which way the model leans.</p><h3>2. When would you report RMSE instead of MAE?</h3><p><strong>What they’re testing:</strong> whether you tie the metric to the cost of errors.</p><p><strong>Strong answer:</strong> MAE treats every unit of error the same; RMSE squares errors first, so large misses dominate. I’d use RMSE when big errors are disproportionately costly, such as a property valuation £100k out, and MAE when costs are roughly linear or the data has outliers I don’t want driving model choice. I’d report both, because the gap is diagnostic. RMSE is always at least MAE. If they’re close, errors are similar in size. If RMSE is much larger, a few large misses are hiding behind the average, and I’d go and find them. MSE itself I’d use for training but rarely report, because its units are squared.</p><p><strong>Red flag:</strong> “RMSE is more accurate.” They measure different things.</p><h3>3. A colleague wants to compare forecasting accuracy across products using MAPE. Any concerns?</h3><p><strong>What they’re testing:</strong> whether you know a metric’s failure modes, not just its formula.</p><p><strong>Strong answer:</strong> MAPE, the mean absolute percentage error, is attractive because it’s scale-free: a 10% miss on a big product and a small one look the same. But it divides by the actual value, so it explodes when actuals are near zero, such as slow-moving products or days with no sales, and it’s undefined at zero. It also penalises over-forecasts and under-forecasts asymmetrically. I’d exclude or handle near-zero actuals explicitly, or use an alternative such as weighted MAPE (total absolute error divided by total actuals) or MAE scaled against a simple benchmark.</p><p><strong>Red flag:</strong> “MAPE is great because it’s a percentage.”</p><h3>4. Walk me through a confusion matrix using a real business example.</h3><p><strong>What they’re testing:</strong> whether you can turn four cells into business consequences.</p><p><strong>Strong answer:</strong> take mortgage fraud detection, with fraud as the positive class. Rows are the actual class, columns the prediction. A <strong>true positive</strong> is a fraudulent application we flagged. A <strong>true negative</strong> is a genuine one we approved. A <strong>false positive</strong> (Type I error) is a genuine applicant we flagged: a delay, extra checks, maybe a lost customer. A <strong>false negative</strong> (Type II error) is fraud we approved: a direct financial loss. The point of the matrix is that those two errors have very different costs, so a single score that blends them hides the most important information. One practical warning: sklearn’s confusion_matrix orders classes by label, so with 0/1 labels the negative class comes first. Always check which corner is which.</p><p><strong>Red flag:</strong> reciting the four definitions without saying what any of them costs.</p><h3>5. Why must sensitivity and specificity be read together?</h3><p><strong>What they’re testing:</strong> whether you understand the trade-off, not just the ratios.</p><p><strong>Strong answer:</strong> sensitivity (recall) is TP / (TP + FN): of the real positives, the share we caught. Specificity is TN / (TN + FP): of the real negatives, the share we correctly left alone. Either one alone can be gamed. Flag everything and sensitivity is a perfect 1 while specificity is 0. Flag nothing and it’s the reverse. Both trivial models are useless. Reported together, they describe the trade-off the model actually makes at its current threshold, and that is the conversation to have with the business.</p><p><strong>Red flag:</strong> “Our sensitivity is 98%” with no specificity attached.</p><h3>6. Fraud is 1% of your data. Your model is 99% accurate. Should we deploy it?</h3><p><strong>What they’re testing:</strong> the accuracy paradox, the classic imbalanced-data trap.</p><p><strong>Strong answer:</strong> I can’t tell from that number, and it’s a warning sign. A model that predicts “not fraud” for everything is also 99% accurate and catches nothing. With a rare class, accuracy is dominated by the majority class. I’d ask for the confusion matrix, then sensitivity and precision for the fraud class. I’d compare against that trivial baseline, and check performance on a stratified or time-based hold-out set so the rare cases are actually represented.</p><p><strong>Red flag:</strong> “99% is excellent, ship it.”</p><h3>7. Precision or recall: which matters more?</h3><p><strong>What they’re testing:</strong> whether you answer with “it depends on what?”, and then say exactly what.</p><p><strong>Strong answer:</strong> recall asks “of all the real positives, how many did we find?”. Precision asks “when we flag something, how often are we right?”. Which matters depends on the cost of each error and on capacity. For cancer screening, missing a case is the expensive error, so I’d favour recall and confirm positives with a second test. For an alert queue reviewed by a small team, a flood of false alarms wastes their time and erodes trust, so precision matters more. When both matter, F1 combines them, but I’d still report the two separately, because F1 hides which one is weak.</p><p><strong>Red flag:</strong> picking one without asking about the use case.</p><h3>8. How would you choose the classification threshold?</h3><p><strong>What they’re testing:</strong> whether you know 0.5 is a default, not a decision.</p><p><strong>Strong answer:</strong> most classifiers output a score, and the threshold converts it into an action. Lowering it raises sensitivity and lowers specificity; raising it does the opposite. I’d choose it on the validation set using the business costs. For example: if a missed fraud costs £5,000 and a false alarm costs £20 of review time, I’d pick the threshold that minimises total expected cost. Where there’s a capacity limit, such as reviewers who can handle 200 cases a day, the threshold is simply whatever produces 200 flags. Then I’d confirm the result on the test set and monitor it, because the best threshold shifts when the share of positives changes.</p><p><strong>Red flag:</strong> “I use 0.5.”</p><h3>9. What is a lift chart, and when is it more useful than accuracy?</h3><p><strong>What they’re testing:</strong> whether you recognise ranking problems.</p><p><strong>Strong answer:</strong> a lift chart, or cumulative gains chart, evaluates how well a model puts the important cases first. Sort cases by the model’s score, highest first, and plot the cumulative number of real positives found against the number of cases checked. Compare with two lines: random selection, a straight diagonal at the average rate, and a perfect model, which finds every positive first. It’s the right tool when you can only act on the top of the list: investigating 300 of 100,000 expense claims, or calling the 5,000 customers most likely to respond. Lift is the ratio at a cut-off: if the top 10% contains 30% of the positives, the lift is 3. For regression the same chart adds up actual values, such as spend, in order of the predictions.</p><p><strong>Red flag:</strong> treating it as a nicer-looking accuracy plot.</p><h3>10. Can a model with a poor MAE still be useful?</h3><p><strong>What they’re testing:</strong> whether you separate how accurate predictions are from how well they rank.</p><p><strong>Strong answer:</strong> yes, if the task is ranking. If a model predicts every customer’s spend at exactly half the true value, its MAE is terrible but the order is perfect, so its lift chart matches the perfect model. If the business only needs “who first?”, that model is ideal. If it needs the actual values, say for budgeting, it’s not, though here a simple recalibration would fix it. The lesson: choose the metric by the decision. Error metrics for “how much?”, the confusion matrix for “which category?”, lift for “which first?”.</p><p><strong>Red flag:</strong> “no, high error means a bad model.”</p><h3>What seniors do differently</h3><p>Look back at the strong answers and the same four habits appear every time:</p><ul><li><strong>They start from the decision.</strong> What happens to the predictions decides the metric, not the other way round.</li><li><strong>They price the mistakes.</strong> A false alarm and a miss are never the same error, so they are never blended into one number without thought.</li><li><strong>They look past the headline number.</strong> The gap between RMSE and MAE, the sign of the mean error, the confusion matrix behind the accuracy.</li><li><strong>They compare against a baseline.</strong> “Always no”, random selection, the average model. A score means nothing until you know what doing nothing would have scored.</li></ul><p>That’s the real difference between a junior and a senior answer. Both know the formulas. Only one knows which formula the business actually needs.</p><p><em>Found this useful? Follow for more interview-focused breakdowns of the ideas that make machine learning work in practice, and pass it to someone preparing for their next ML interview.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07e6806c9684" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/10-machine-learning-interview-questions-on-model-evaluation-that-separate-juniors-from-seniors-07e6806c9684">10 Machine Learning Interview Questions on Model Evaluation That Separate Juniors From Seniors</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Metric Is the Model: How Choosing How You Score Decides What You Build]]></title>
            <link>https://medium.com/the-learning-rate/the-metric-is-the-model-how-choosing-how-you-score-decides-what-you-build-0697f57e9169?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/0697f57e9169</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 22:23:25 GMT</pubDate>
            <atom:updated>2026-11-15T22:23:25.297Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why the same model can look great or useless depending on the number you report, and how to pick the one that matches the decision.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VwJ4yK3kP34tgG_c1a2big.png" /></figure><p>Two models value the same ten houses.</p><p>Model A is off by £10k on every single house. Model B nails nine of them exactly and misses the tenth by £60k.</p><p>Which is better? Ask the mean absolute error and Model B wins easily: an average miss of £6k against £10k. Ask the root mean squared error and Model A wins: £10k against about £19k.</p><p>Same data. Same predictions. Opposite verdicts.</p><p>This is the part of machine learning that rarely makes the tutorials. The metric isn’t a neutral scoreboard you check at the end. It’s a statement about <strong>which mistakes you can live with</strong>. Choose it carelessly and you’ll optimise, select and ship a model that solves the wrong problem, with a great number on the slide.</p><h3>1. Average miss or worst miss? Decide before you measure</h3><p>For numeric predictions, the two everyday metrics ask different questions.</p><p><strong>MAE (mean absolute error)</strong> is the average size of a miss. Every pound of error counts the same. It’s easy to explain (“we’re typically £11k out”) and a few extreme cases can’t dominate it.</p><p><strong>RMSE (root mean squared error)</strong> squares each error before averaging. A miss of 2 counts as 4; a miss of 10 counts as 100. Big misses dominate, so RMSE is the metric of someone who is afraid of disasters.</p><p>Neither is “more correct”. The question is what a large error costs you. If a valuation that is £100k out triggers a bad loan, one big miss is far worse than ten small ones, and RMSE is the honest score. If you’re forecasting footfall and every unit of error costs the same, MAE tells the truth and RMSE overreacts to a few odd days.</p><p>There’s a free diagnostic hiding here. RMSE is always at least as big as MAE. <strong>When they’re close, your errors are all of similar size. When RMSE pulls well ahead, a handful of large misses is hiding behind a respectable average.</strong> Always report both, and look at the gap.</p><h3>The number almost nobody reports</h3><p><strong>Mean error (ME)</strong> is the average error with the signs kept. It can be tiny while the model is terrible, because errors of +100 and −100 cancel out. So why bother?</p><p>Because it detects <strong>bias</strong>. Imagine a model whose errors are +0.2, +0.2, +0.1, +0.3, +0.2. Its MAE looks fine. But every error points the same way: it always under-predicts. That’s not noise, that’s a systematic fault, and it’s the easiest kind to fix. Add 0.2 to every prediction and the typical error collapses from 0.20 to 0.04.</p><p>In production, a mean error that drifts away from zero is often your first sign that the world has changed under the model.</p><h3>2. The 99% accurate model that catches nothing</h3><p>Now switch to predicting a category, say fraud or not fraud. The natural metric is accuracy: the share of predictions that are right.</p><p>Suppose 1 transaction in 100 is fraud. Here is a model that takes no time to build: it says “not fraud” to everything. On 1,000 transactions it is right 990 times. <strong>Accuracy: 99%.</strong> Frauds caught: zero.</p><p>This is the accuracy paradox, and it shows up wherever the thing you care about is rare: fraud, disease, loan defaults, equipment failures. Accuracy is dominated by the big, boring class, so it rewards a model for ignoring the very cases it was built to find.</p><p>The fix is to stop compressing everything into one number too early. Every prediction lands in one of four boxes:</p><ul><li><strong>True positive:</strong> flagged, and it really was fraud.</li><li><strong>False positive:</strong> flagged, but it was genuine. A false alarm.</li><li><strong>False negative:</strong> not flagged, but it was fraud. A miss.</li><li><strong>True negative:</strong> not flagged, and it was genuine.</li></ul><p>Laid out as a 2×2 table, this is the <strong>confusion matrix</strong>. It’s the most underrated exhibit in machine learning, because it shows <em>which kind</em> of wrong the model is. That’s the only thing a business cares about. A false alarm annoys a genuine customer. A miss loses money. They are not the same mistake, and accuracy treats them as if they were.</p><h3>3. The threshold is a business decision, not a technical one</h3><p>Two numbers from the confusion matrix capture the trade-off:</p><ul><li><strong>Sensitivity (recall):</strong> of the real frauds, what share did we catch?</li><li><strong>Specificity:</strong> of the genuine transactions, what share did we correctly leave alone?</li></ul><p>Most classifiers don’t output “fraud” or “not fraud”. They output a score, and <em>you</em> pick a cut-off. Lower the cut-off and you catch more fraud (sensitivity rises), but you also flag more genuine customers (specificity falls). Raise it and the opposite happens. There is no setting that maximises both.</p><p>That’s why these two numbers only make sense as a pair. A model that flags <em>everything</em> has perfect sensitivity. A model that flags <em>nothing</em> has perfect specificity. Each looks flawless on one number and is useless in practice.</p><p>So where should the cut-off go? Not where the data scientist finds it elegant. It goes where the <strong>costs</strong> balance:</p><ul><li>Screening for a serious, treatable disease? A miss can be fatal, a false alarm means a follow-up test. Push sensitivity up.</li><li>Blocking card payments? Every false alarm is a customer stuck at a till, and possibly a lost customer. Protect specificity.</li><li>Choosing which applications get a manual review? The limit is how many reviewers you have, which leads to the next idea.</li></ul><p>The most useful conversation you can have with a stakeholder isn’t “the model is 94% accurate”. It’s: <strong>“At this setting we catch 8 in 10 frauds and wrongly flag 1 in 50 genuine customers. Do you want to move that line?”</strong></p><h3>4. When you can only act on the top of the list</h3><p>Here is a situation every operations team knows. There are 100,000 expense claims, and investigators have time for 300. Nobody will act on the other 99,700 predictions, so their accuracy is irrelevant. The only question that matters is: <strong>are the most suspicious claims at the top of the list?</strong></p><p>That’s a <strong>ranking</strong> problem, and it needs its own yardstick: the <strong>lift chart</strong>, also called a cumulative gains chart. Sort the cases by the model’s score, highest first, then walk down the list counting the real hits you’ve found so far. Plot that running count, and add two reference lines:</p><ul><li><strong>Random:</strong> checking cases in random order finds hits at the average rate, a straight diagonal.</li><li><strong>Perfect:</strong> every real hit comes first, a steep climb that then goes flat.</li></ul><p>Your model sits somewhere between the two. The question it answers is concrete: “If we review the top 300, how many frauds do we find, compared with picking 300 at random?” If the model’s top 10% contains 30% of all fraud, it has a <strong>lift of 3</strong>. Your investigators are three times as productive as they would be by guessing.</p><p>The same idea works for numbers. Sort customers by predicted spend, add up their actual spend as you go, and see how quickly the curve climbs compared with the average.</p><p>Here’s the twist that surprises people. <strong>A model can rank perfectly while being badly wrong on every value.</strong> Predict every customer’s spend at exactly half its true value and the MAE is awful, but the order, and the lift chart, are unchanged. When the decision is “which ones first?”, error metrics are measuring the wrong thing.</p><h3>How to choose: start from the decision, not the dashboard</h3><p>Before you train anything, write down what will happen to the predictions. The metric follows from that:</p><ul><li><strong>Every prediction is used, and every unit of error costs the same</strong> (forecasting volumes, estimating durations): MAE, in the target’s own units.</li><li><strong>Every prediction is used, and big misses are disproportionately costly</strong> (valuations, risk limits): RMSE, and check the gap to MAE.</li><li><strong>You need to know if the model leans one way:</strong> mean error, tracked over time.</li><li><strong>A yes/no decision where both mistakes cost the same and classes are balanced:</strong> accuracy is fine.</li><li><strong>A yes/no decision where the classes are imbalanced or the mistakes cost differently:</strong> the confusion matrix, sensitivity and specificity, and a threshold set by the business cost of each error.</li><li><strong>You will only act on the top of a list:</strong> a lift chart at your real capacity.</li></ul><p>One rule holds across all of them: measure on data the model hasn’t seen, and compare models on the same data.</p><h3>The takeaway</h3><p>A metric quietly encodes a value judgement: which mistakes matter, and how much. When the team optimises it, compares models on it and celebrates it, that judgement shapes the model as much as any algorithm does.</p><p>So choose it deliberately. Ask what each kind of error costs, who pays for it, and what you’ll actually do with the predictions. Get that right, and the metric becomes the clearest specification your model will ever have. Get it wrong, and you’ll build an excellent answer to a question nobody asked.</p><p><em>If this changed how you look at a model score, follow for more on the judgement side of machine learning: the decisions that matter once a model leaves the notebook.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0697f57e9169" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/the-metric-is-the-model-how-choosing-how-you-score-decides-what-you-build-0697f57e9169">The Metric Is the Model: How Choosing How You Score Decides What You Build</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Machine Learning Interview Questions About Whether Your Model Will Generalise]]></title>
            <link>https://medium.com/the-learning-rate/10-machine-learning-interview-questions-about-whether-your-model-will-generalise-ed762233bb76?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed762233bb76</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[interview-questions]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 22:12:34 GMT</pubDate>
            <atom:updated>2026-11-15T22:12:34.216Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What separates candidates who can train a model from those who can trust one.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFiRAyD4huPMgXPb72NehA.png" /></figure><p>Almost anyone can call .fit() and print an accuracy score. What interviewers really want to know is whether you can tell when that score is lying.</p><p>That comes down to one question: <strong>will this model generalise?</strong> Will it work on data it has never seen, in a world that keeps changing?</p><p>Here are ten questions that probe exactly that. For each one you’ll find what the interviewer is really testing, a strong answer, and the red flag that quietly sinks candidates.</p><h3>1. Your model scores 99% on training data and 72% on new data. What’s going on?</h3><p><strong>What they’re testing:</strong> whether you recognise overfitting and know what to do about it.</p><p><strong>Strong answer:</strong> the gap is the story. The model has learned the noise and quirks of the training data, not just the pattern, so it’s overfitting. I’d first check the evaluation is clean (no leakage, same distribution). Then I’d reduce the model’s flexibility: a simpler model, regularisation, fewer features, shallower trees or early stopping. Or I’d give it more data. I’d judge every fix by validation error, not training error.</p><p><strong>Red flag:</strong> “99% is great, so it’s nearly there.” Training accuracy on its own proves the model can remember, not that it can predict.</p><h3>2. What is the no free lunch theorem, and why should a practitioner care?</h3><p><strong>What they’re testing:</strong> whether you understand that learning always rests on assumptions.</p><p><strong>Strong answer:</strong> averaged over every possible problem, no learning algorithm beats any other. Data alone says nothing about unseen cases. With 3 yes/no inputs there are 8 possible cases and 256 possible rules. Seeing 5 cases leaves 8 rules that all fit, and they disagree on every unseen case. An algorithm only works by assuming something about the problem: smoothness, similar inputs give similar outputs, the future resembles the past. In practice that means there is no universally best model. You choose one whose assumptions fit your problem, and compare candidates empirically.</p><p><strong>Red flag:</strong> naming a favourite algorithm as “always the best”.</p><h3>3. Why do we need training, validation and test sets? Isn’t a train/test split enough?</h3><p><strong>What they’re testing:</strong> whether you know that choosing a model uses up data.</p><p><strong>Strong answer:</strong> each set has one job. The training set fits the model’s parameters. The validation set compares models and tunes settings. The test set gives one final, unbiased estimate of performance. With only two sets, the test set ends up doing the validation job: you pick the model that scored best on it, and then that score is no longer an honest estimate. Typical splits are 70/15/15 or 60/20/20. With very large data the validation and test shares can be much smaller, since 1% of ten million rows is still 100,000 examples. With small data, use cross-validation for choosing and keep a final hold-out for reporting.</p><p><strong>Red flag:</strong> “I tune on the test set because it’s the most realistic data.”</p><h3>4. You compared 20 models and the best scored 91% on validation. What will it score in production?</h3><p><strong>What they’re testing:</strong> whether you know the winner’s curse.</p><p><strong>Strong answer:</strong> probably less than 91%. The winner was picked <em>because</em> it scored highest, and part of any top score is luck on those particular validation cases. Imagine 50 “models” that guess at random, each truly 50% accurate. On 100 validation cases the luckiest typically scores around 60%. Test it on fresh data and it drops back to 50%. Real models show the same bias, just smaller. That’s why I’d report performance on an untouched test set, used once, after the choice is made.</p><p><strong>Red flag:</strong> quoting the best validation score as the expected production performance.</p><h3>5. Explain underfitting and overfitting. How do you tell them apart?</h3><p><strong>What they’re testing:</strong> whether you can diagnose from evidence, not just define terms.</p><p><strong>Strong answer:</strong> underfitting means the model is too simple to capture the pattern (high bias): error is high on both training and validation data. Overfitting means it is flexible enough to fit the noise (high variance): training error is low, validation error much higher. As complexity rises, training error keeps falling. Validation error falls, bottoms out, then rises. I’d plot both against complexity, or against training-set size (a learning curve), and choose the point where validation error is lowest. For underfitting I’d add flexibility or better features. For overfitting I’d simplify, regularise or add data.</p><p><strong>Red flag:</strong> “more complex models are always more accurate”.</p><h3>6. What is VC dimension, in plain words?</h3><p><strong>What they’re testing:</strong> whether you understand model capacity beyond counting parameters.</p><p><strong>Strong answer:</strong> it measures how flexible a family of models is. It is the largest number of points the family can <em>shatter</em>: label correctly under every possible yes/no assignment. A threshold on a line has VC dimension 1. An interval on a line has 2. A straight line in a 2D plane has 3: it can separate any labelling of 3 points, but not every labelling of 4. Why it matters: the higher the VC dimension, the more data you need before a good training score says anything about new data. It is the theory behind “flexible models need more data”.</p><p><strong>Red flag:</strong> “it’s the number of parameters”. Often related, but not the same thing.</p><h3>7. How do you know if you have enough data?</h3><p><strong>What they’re testing:</strong> whether you link data volume to model flexibility and to the guarantee you want.</p><p><strong>Strong answer:</strong> theory says the data you need grows with three things: how flexible the model family is, how small an error gap you want to guarantee, and how confident you want to be. The more candidate models the algorithm can choose from, the higher the chance one fits by luck, so more data is needed to rule that out. Each halving of the error gap you’ll accept costs more data again. In practice I’d draw a learning curve: train on 10%, 20% and so on up to 100% of the data and plot validation error. If it is still falling steeply, more data will help. If it has flattened, the gain will come from features or a different model instead.</p><p><strong>Red flag:</strong> a fixed rule of thumb like “10,000 rows is always enough”.</p><h3>8. You tuned 200 hyperparameter combinations with cross-validation. Can you trust the best CV score?</h3><p><strong>What they’re testing:</strong> whether you see that heavy tuning is model selection, and selection causes optimism.</p><p><strong>Strong answer:</strong> not fully. Each combination is effectively a different model, so picking the best of 200 brings back the winner’s curse: the top score is partly noise that happened to suit those folds. The more combinations I try, the more optimistic it gets. To get an honest number I’d evaluate the final configuration once on a held-out test set that took no part in the tuning, or use nested cross-validation, with an outer loop for estimating and an inner loop for tuning. I’d also be wary when the winning settings sit at the edge of the search range, or when CV scores vary a lot between folds.</p><p><strong>Red flag:</strong> reporting the best grid-search score as the final result.</p><h3>9. Your validation results look too good to be true. What would you check?</h3><p><strong>What they’re testing:</strong> whether you can spot data leakage, the most common reason evaluations lie.</p><p><strong>Strong answer:</strong> I’d look for ways the answer reached the model before the exam. Preprocessing fitted on the whole data set: scalers, imputers or encoders must be fitted on training data only, ideally inside a pipeline. A random split on time-ordered data: the model trains on the future and is tested on the past, so I’d split by time instead. The same customer, account or patient in both training and test: I’d split by group. Features that are only known after the outcome, such as a “claim settled” flag used to predict whether a claim is fraud. Duplicates across the split. And for rare classes, I’d stratify so every split has positives.</p><p><strong>Red flag:</strong> “It’s a strong model, so the score is plausible”, with no investigation.</p><h3>10. Your model performed well at launch, but six months later its accuracy has dropped. No code changed. Why?</h3><p><strong>What they’re testing:</strong> whether you know the assumption that every generalisation guarantee depends on.</p><p><strong>Strong answer:</strong> all of the theory, and every honest test score, assumes new data comes from the same distribution as the training data. When the world changes, that assumption breaks. The inputs can shift (new customer segments, a new product channel): data drift. The relationship between inputs and outcome can change too (fraudsters adapt, interest rates move behaviour): concept drift. The test score was true for the old world. I’d monitor the input distributions and the live performance against a baseline, set alert thresholds, and plan regular retraining on recent data, with a fresh time-based hold-out each time.</p><p><strong>Red flag:</strong> “the model must have a bug”, or “just retrain it” with no plan to detect the next shift.</p><h3>The pattern behind every strong answer</h3><p>Read the ten strong answers again and the same habit shows up each time. The candidate never takes a score at face value. They ask:</p><ul><li><strong>Was it measured on data the model never saw?</strong></li><li><strong>Was that data used to make any choices?</strong></li><li><strong>Is the model more flexible than the data can support?</strong></li><li><strong>Will the future look like the data it was tested on?</strong></li></ul><p>That’s what interviewers mean by “ML maturity”. Training a model is a skill you can learn in an afternoon. Knowing when to trust it is what gets you hired, and what keeps a model working long after the demo.</p><p><em>Found this useful? Follow for more interview-focused breakdowns of the ideas that make machine learning work in the real world, and share it with someone preparing for their next ML interview.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed762233bb76" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/10-machine-learning-interview-questions-about-whether-your-model-will-generalise-ed762233bb76">10 Machine Learning Interview Questions About Whether Your Model Will Generalise</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Model Aced the Test. That Proves Less Than You Think.]]></title>
            <link>https://medium.com/the-learning-rate/your-model-aced-the-test-that-proves-less-than-you-think-441132165dc2?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/441132165dc2</guid>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 22:07:26 GMT</pubDate>
            <atom:updated>2026-11-15T22:07:26.232Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why learning from data works at all, and the quiet ways it fails: overfitting, peeking at the test set, and crowning the “best” model.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mw-3WI50UtSeGxuxizxFiA.png" /></figure><p>The model scored 97% on the hold-out set. The team celebrated, wrote the slide, and shipped it.</p><p>Three months later, it was performing at 81% in production. Nobody had touched the code. Nobody had made a mistake you could point to in a review.</p><p>This story is so common it barely registers as a failure any more. But it isn’t bad luck. It’s what happens when we treat a test score as proof, rather than as what it really is: a <strong>noisy estimate built on a set of assumptions</strong>, some of which we quietly broke along the way.</p><p>Here are five ideas that change how you read any model score, including your own.</p><h3>1. Every model is a bet that tomorrow looks like yesterday</h3><p>Here is an uncomfortable fact: data alone cannot tell you anything about cases you haven’t seen.</p><p>Take the smallest possible problem. Three yes/no inputs give 8 possible cases. Any rule that maps those 8 cases to yes or no is a possible “truth”, and there are 256 such rules. Now suppose you’ve seen 5 of the 8 cases, labelled. Exactly 8 rules agree with everything you’ve seen, and they disagree with each other on every one of the 3 cases you haven’t. Without some extra assumption, every guess about those 3 is as good as any other.</p><p>That is the <strong>no free lunch</strong> idea in miniature. Learning only works because we add assumptions: the pattern is smooth, similar inputs give similar outputs, and, above all, <strong>the future comes from the same process as the past</strong>.</p><p>That last assumption is the one that breaks most often in real life. Interest rates move. A new fraud scheme appears. A product changes who your customers are. Your model didn’t get worse; the world stopped matching the bet it was built on.</p><p><strong>The judgement:</strong> before asking “how accurate is the model?”, ask “what would have to stay the same for this score to hold?”. Then watch those things after you deploy.</p><h3>2. A training score is a memory test, not an exam</h3><p>A flexible enough model can score 100% on its training data by memorising it: every quirk, every typo, every bit of random noise.</p><p>Picture noisy points that follow a gentle curve. A straight line misses the curve: it is too simple, so it does badly everywhere. That’s <strong>underfitting</strong>. A 15th-degree polynomial threads through every single point, wiggling wildly in between. Its training error is close to zero. On new points, it’s worse than the simple curve, because it learned the noise as if it were the pattern. That’s <strong>overfitting</strong>.</p><p>The pattern is always the same. As a model gets more flexible, training error keeps falling. Error on unseen data falls at first, then turns and rises. The sweet spot sits at the bottom of that second curve, and you can only find it by measuring on data the model never trained on.</p><p><strong>The judgement:</strong> a model that is suspiciously perfect on training data hasn’t proved it’s good. It has proved it can remember. Only unseen data tells you whether it learned.</p><h3>3. The more a model can say, the less its fit proves</h3><p>Why should a simpler model be more trustworthy? Not because simplicity is elegant, but because of a counting argument.</p><p>Imagine a lottery. If you buy one ticket and win, something unusual happened. If you buy a million tickets and one wins, that’s exactly what you’d expect. Training a model works the same way. A learning algorithm searches through many possible models and picks the one that fits your data best. The more models it can choose from, the more likely it is that <strong>one of them fits well purely by chance</strong>.</p><p>Learning theory makes this precise. The guarantees about how well a model will do on new data get weaker as the set of models you search gets larger, and the only way to win them back is more data. For infinitely flexible model families, theory measures flexibility by the <strong>VC dimension</strong>: the largest number of points the family can label in every possible way. A straight line in two dimensions can separate any labelling of 3 points, but not of every 4, so its VC dimension is 3. The higher this number, the more data you need before a good fit means anything.</p><p><strong>The judgement:</strong> a complex model on a small data set is buying lots of lottery tickets. When a simple model and a complex one score about the same, the simple one’s score is the one you can believe.</p><h3>4. The winner’s curse: your best model is flattered</h3><p>So you do the right thing. You split off a validation set, train ten candidate models, and choose the one with the best validation score. Surely that score is now an honest estimate?</p><p>It isn’t, and a thought experiment shows why. Take 50 “models” that are pure coin flips: each one guesses yes or no at random, so each is truly 50% accurate. Score all 50 on the same 100 validation cases. They won’t all score 50%. Some get lucky, and the best of the 50 typically scores <strong>around 60%</strong>. Pick that one, test it on fresh data, and it falls straight back to 50%.</p><p>Real models aren’t coin flips, but the same effect is always there. The winner was chosen <em>because</em> it scored highest, and part of any high score is luck on those particular cases. The more models and settings you compare, the bigger the flattering.</p><p><strong>The judgement:</strong> the moment you use a data set to make a choice, its score stops being an unbiased estimate. That’s why serious work uses three splits: <strong>training</strong> to fit, <strong>validation</strong> to choose, and a <strong>test</strong> set that played no part in either, to report.</p><h3>5. Every peek spends the test set</h3><p>The test set is the most valuable data you own, and it has an odd property: it wears out when you look at it.</p><p>The first time you score a model on it, you get an honest estimate. But suppose the result disappoints, so you tweak a feature and test again. Then adjust a threshold and test again. Each change is small and well-meant. After twenty rounds, you have done exactly what the validation set was for: you have chosen a model <em>based on the test score</em>. The winner’s curse is back, and your “untouched” test set has quietly become a second validation set.</p><p>The same leak happens in subtler ways. Scaling or imputing with statistics calculated on the whole data set, test rows included. Splitting time-ordered data at random, so the model trains on next month and is tested on last month. Letting near-duplicate records land on both sides of the split. Each one lets a little of the answer into the exam room.</p><p><strong>The judgement:</strong> treat the test set like a sealed envelope. Fit every preprocessing step on training data only, split by time when time matters, and open the envelope once. If you have to open it again, the honest thing is to say so, or to collect fresh data.</p><h3>Five questions to ask before you trust a score</h3><p>Next time someone shows you a model’s accuracy, including you, ask:</p><ol><li><strong>What has to stay the same for this to hold?</strong> Which assumptions about the data, the customers or the market is this score betting on?</li><li><strong>Was it measured on data the model never trained on?</strong> A training score is a memory test.</li><li><strong>How flexible is the model compared with the amount of data?</strong> A complex model on a small sample should make you more sceptical, not less.</li><li><strong>How many models were compared to find this one?</strong> The more candidates, the more the winner is flattered.</li><li><strong>How many times has the test set been looked at?</strong> If the answer is “we’ve been iterating on it”, it’s no longer a test.</li></ol><h3>The takeaway</h3><p>Machine learning is often sold as “let the data speak”. But data never speaks about the cases you haven’t seen. What lets a model generalise is a chain of assumptions and disciplines: the future resembles the past, the model is no more flexible than the data can support, the choice was made on one data set and judged on another.</p><p>A great score is not the end of the conversation. It’s an invitation to ask how it was earned.</p><p><em>If this changed how you read a model score, follow for more on the judgement side of machine learning: the questions that decide whether a model works once it leaves the notebook.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=441132165dc2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/your-model-aced-the-test-that-proves-less-than-you-think-441132165dc2">Your Model Aced the Test. That Proves Less Than You Think.</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 statistics interview questions for data science and ML, and what interviewers actually listen…]]></title>
            <link>https://medium.com/the-learning-rate/10-statistics-interview-questions-for-data-science-and-ml-and-what-interviewers-actually-listen-b889a78b70a6?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b889a78b70a6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 00:01:18 GMT</pubDate>
            <atom:updated>2026-11-02T00:01:18.160Z</atom:updated>
            <content:encoded><![CDATA[<h3>10 statistics interview questions for data science and ML, and what interviewers actually listen for</h3><p><em>MLE, outliers, correlation and bootstrapping: the traps that catch most candidates</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ud8LYSdO6fyQ1UOHs9R0Mw.png" /></figure><p>Statistics questions are where data science interviews get honest. Anyone can call .mean() or fit a regression in two lines of Python. Interviewers want to know whether you understand what those numbers mean, when they mislead, and how far you’d trust them before making a decision.</p><p>The questions below cover four topics that come up again and again: maximum likelihood estimation, outliers, regression and correlation, and bootstrapping. Each answer has three parts: what a strong answer covers, what the interviewer is really listening for, and the answer that quietly costs you the role.</p><h3>1. “What is maximum likelihood estimation? Explain it with a simple example.”</h3><p><strong>A strong answer:</strong> MLE picks the parameter values that make the data you actually observed most likely. Take a coin with unknown probability of heads θ. You flip it 200 times and see 138 heads. For each possible θ you can ask how likely exactly that result would be, θ¹³⁸ × (1 − θ)⁶², and choose the θ that makes it largest. For a coin the answer is simply the share of heads, θ̂ = 138 ÷ 200 = 0.69. The same principle scales to models with thousands of parameters.</p><p><strong>What they’re listening for:</strong> The idea of comparing candidate parameters by how well they explain the observed data, and a concrete example where the answer is intuitive.</p><p><strong>The answer that costs you:</strong> “It finds the most likely parameter.” Close, but backwards: it finds the parameter under which the <em>data</em> is most likely. That distinction matters later when Bayesian methods come up.</p><h3>2. “Why do we maximise the log-likelihood, and how does MLE relate to the loss functions used in machine learning?”</h3><p><strong>A strong answer:</strong> The likelihood multiplies one probability per data point, all below 1. With thousands of points the product becomes so small that a computer rounds it to zero, which is arithmetic underflow. Taking the log turns the product into a sum, which is stable to compute and easy to differentiate, and because log is increasing, the maximum is in the same place. Most optimisers minimise rather than maximise, so in practice we minimise the negative log-likelihood. Many familiar losses are exactly that: cross-entropy for classification is the negative log-likelihood of the true labels, and least squares is the negative log-likelihood when errors are independent and normally distributed.</p><p><strong>What they’re listening for:</strong> Underflow as the practical reason, the fact that the maximum doesn’t move, and the link from MLE to cross-entropy and least squares.</p><p><strong>The answer that costs you:</strong> “Logs make the maths easier,” with nothing about numerical stability or the connection to everyday loss functions.</p><h3>3. “Your estimate says a business decision will be profitable. What else do you check before recommending it?”</h3><p><strong>A strong answer:</strong> How uncertain the estimate is, and how close it sits to the break-even point. A worked example: a bet pays +60 on heads and −120 on tails, so it only profits if θ is above 0.667. From 200 flips the MLE is 0.69, which suggests playing. But the standard error is about √(0.69 × 0.31 ÷ 200) ≈ 0.033, so 0.69 is less than one standard error above break-even, and values below 0.667 remain entirely plausible. I’d report the estimate with an interval, show how often the decision would be wrong across that interval, and if the stakes justify it, gather more data before committing.</p><p><strong>What they’re listening for:</strong> Treating an estimate as uncertain, comparing that uncertainty with the decision threshold, and turning statistics into a recommendation.</p><p><strong>The answer that costs you:</strong> “The expected value is positive, so go ahead.” A best guess isn’t a decision.</p><h3>4. “When would you report the median instead of the mean?”</h3><p><strong>A strong answer:</strong> When the data is skewed or contains outliers. The mean uses every value’s size, so a few extreme values drag it; the median only cares about order, so it barely moves. Nine test scores between 70 and 90 average 80, and adding one score of 20 drops the mean to 74 while the median moves only from 80 to 79. That’s why house prices, salaries and transaction amounts are usually reported as medians. I’d often report both: a big gap between them is itself a signal that the distribution is skewed or has extreme values worth investigating.</p><p><strong>What they’re listening for:</strong> Robustness as the reason, a quick example, and the habit of comparing the two.</p><p><strong>The answer that costs you:</strong> “The median is better because it’s in the middle.” Neither measure is always better; it depends on the data and the question.</p><h3>5. “How do you detect outliers? When might two methods disagree?”</h3><p><strong>A strong answer:</strong> Start by plotting: histograms, box plots, scatter plots. Then two common rules. The z-score flags values more than about three standard deviations from the mean. The interquartile range (IQR) rule flags values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR. They disagree because the z-score is built from the mean and standard deviation, which outliers themselves inflate. One huge value can stretch the standard deviation enough to hide other extreme values; that’s called masking. The IQR is built from the middle half of the data, so it’s more robust. But neither knows what’s physically possible: an IQR lower fence can be negative and let through a value that’s clearly impossible. For many dimensions at once, methods like isolation forests or local outlier factor help.</p><p><strong>What they’re listening for:</strong> Visual checks first, both rules, the masking problem, and the limits of any purely statistical rule.</p><p><strong>The answer that costs you:</strong> “Anything with |z| &gt; 3 is an outlier,” applied as a universal rule.</p><h3>6. “You’ve found some outliers. Should you remove them?”</h3><p><strong>A strong answer:</strong> Not until I know why they’re there. An outlier is one of three things: an error (a typo, a faulty sensor, an impossible value), a rare but genuine value (a legitimate £4 million mortgage), or a signal (fraud, a failing machine, a security breach). Errors get fixed or removed. Genuine extremes usually stay, perhaps with a log transform or capping so they don’t dominate. Signals must stay, because they may be exactly what the model needs to learn. In fraud detection, deleting outliers before training can remove much of the fraud itself. Whatever I decide, I keep the raw data, document the rule, and compare results with and without the outliers.</p><p><strong>What they’re listening for:</strong> Diagnosing before treating, the three-way distinction, domain knowledge, and awareness that outliers can be the most valuable rows.</p><p><strong>The answer that costs you:</strong> “Yes, they’re noise,” or the opposite extreme, “Never, all data is sacred.”</p><h3>7. “What’s the difference between regression and correlation?”</h3><p><strong>A strong answer:</strong> Both measure how two variables move together, but they answer different questions. Regression finds the best-fitting line, Y = aX + b, for predicting Y from X. “Best” means least squares: the line that minimises the sum of squared vertical distances from the points. The slope is in real units, such as pounds of price per square metre. Correlation, r, measures how tightly the points follow a straight line, on a unitless scale from −1 to +1. The two are linked: slope = r × (standard deviation of Y ÷ standard deviation of X). One more detail interviewers like: regression isn’t symmetric. Regressing Y on X and X on Y give different lines, because each minimises errors in a different direction. Correlation is symmetric.</p><p><strong>What they’re listening for:</strong> Prediction versus strength of relationship, least squares explained in plain words, units versus no units, and the asymmetry.</p><p><strong>The answer that costs you:</strong> Treating them as the same thing, or being unable to say what “best fit” means.</p><h3>8. “Two variables have a correlation of 0. Does that mean they’re unrelated? And what if r is 0.9?”</h3><p><strong>A strong answer:</strong> r only measures <em>linear</em> relationships. A perfect U-shape or V-shape can have r = 0 while the variables are strongly related, which is why you always plot the data first. A high r of 0.9 says the points lie close to a straight line, but it doesn’t mean X causes Y: both might be driven by a third factor, as ice-cream sales and drownings both rise in summer. The relationship may also not hold outside the range you measured. And correlation from a small sample can be luck: with 50 points, two completely unrelated variables can show r = 0.3 by chance, so I’d check its uncertainty, for example with a bootstrap interval, before relying on it. In ML, very high correlation between two features often means one is redundant.</p><p><strong>What they’re listening for:</strong> Linear only, plot first, correlation isn’t causation, small-sample noise, and the practical use in feature selection.</p><p><strong>The answer that costs you:</strong> “r = 0 means no relationship and r = 0.9 means X drives Y.”</p><h3>9. “You’re asked what drives system failures and given a table of only the failures. What do you do?”</h3><p><strong>A strong answer:</strong> Ask for the non-failures before analysing anything. If you only study cases where the outcome happened, you can’t see what makes it happen, because there’s nothing to compare against. A feature that appears in 80% of failures means nothing if it also appears in 80% of successful runs. The classic example is the Challenger disaster: looking only at the seven flights with O-ring damage showed no link with temperature, but adding the 16 undamaged flights, all launched in warmer weather, revealed a clear negative correlation of about −0.5. The same trap appears in fraud analysis on investigated cases only, churn analysis on leavers only, and pipelines with a WHERE status = &#39;failed&#39; filter. I’d also check whether the decision in question lies outside the range of the data.</p><p><strong>What they’re listening for:</strong> Recognising selection on the outcome, insisting on a comparison group, and a memorable example.</p><p><strong>The answer that costs you:</strong> Diving straight into the failure table and reporting “the most common features of failures” as their causes.</p><h3>10. “What is bootstrapping, and when would you use it?”</h3><p><strong>A strong answer:</strong> It estimates the uncertainty of any statistic from a single sample. Draw a new sample of the same size from your data <em>with replacement</em>, so some rows repeat and others are left out, recompute the statistic, and repeat around 1,000 times. The spread of those results estimates the standard error, and the 2.5th and 97.5th percentiles give a 95% confidence interval. It’s most useful when there’s no simple formula: a median, a correlation, a model’s accuracy on a test set, or an optimal portfolio weight. It also underpins bagging and random forests, where each model is trained on a different bootstrap sample. Its limits: it can’t fix a biased sample, it’s weak for very small samples and extremes like the maximum, and for time series you need to resample blocks of consecutive points to keep the time structure.</p><p><strong>What they’re listening for:</strong> “With replacement”, how the interval is read, why it beats formulas for awkward statistics, the link to ensembles, and its limits.</p><p><strong>The answer that costs you:</strong> “It creates more data,” which misses the point: it creates no new information, only an honest estimate of how much your existing information can be trusted.</p><h3>The pattern behind every strong answer</h3><p>Across all ten questions, the strongest answers share the same habits:</p><ul><li><strong>Every estimate comes with error bars.</strong> A number without its uncertainty is half an answer, whether it’s a coin’s bias, a correlation or a model’s accuracy.</li><li><strong>Diagnose before you treat.</strong> Ask why an outlier exists, or why a table contains only certain rows, before deleting or analysing anything.</li><li><strong>Plot before you summarise.</strong> One chart can expose a curved relationship, a hidden outlier or a missing comparison group that a single number hides.</li><li><strong>Know the limits of each tool.</strong> z-scores can be masked, correlation is linear only, bootstrapping can’t fix a biased sample.</li><li><strong>Finish with a decision.</strong> Statistics matters because of what it changes: whether to launch, whether to bet, which rows to keep.</li></ul><p>The formulas are easy to look up. What interviewers are really testing is whether you know how far to trust them.</p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b889a78b70a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/10-statistics-interview-questions-for-data-science-and-ml-and-what-interviewers-actually-listen-b889a78b70a6">10 statistics interview questions for data science and ML, and what interviewers actually listen…</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t delete your outliers yet. In fraud, they’re the whole point.]]></title>
            <link>https://medium.com/the-learning-rate/dont-delete-your-outliers-yet-in-fraud-they-re-the-whole-point-6a89a803a097?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a89a803a097</guid>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 22:59:22 GMT</pubDate>
            <atom:updated>2026-11-01T22:59:22.755Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How to tell an error from a rare truth before you clean your data</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8v1XbxzRULlS8l802eZ5Zg.png" /></figure><p>There’s a line of code that appears in countless data-cleaning tutorials. Compute each value’s z-score, the number of standard deviations it sits from the average, and drop anything beyond three. It’s quick, it’s tidy, and your charts instantly look better.</p><p>Now run it on a table of mortgage applications before training a fraud model. The loan that’s ten times the applicant’s declared income. The cluster of applications with identical employer details. The account that suddenly moves its entire balance at 3am. Out they go, filed as noise.</p><p>Those rows weren’t noise. In many cases they were the fraud: the exact behaviour the model was supposed to learn to recognise. The cleaning step didn’t improve the data. It removed the answer.</p><p>Outliers aren’t a single thing to be cleaned away. Some are mistakes, some are rare but real, and some are the most important rows you have. The skill is telling which is which before you hit delete.</p><h3>Three kinds of strange</h3><p>Every outlier is one of three things, and each needs a different response.</p><p><strong>1. An error.</strong> A typo, a faulty sensor, a misplaced decimal point, a test record left in production. A human height of 812 cm, an income of £9,999,999 typed into a form, a guess of 23 lbs for the weight of an adult elephant. These values describe nothing real. Fix them if you can, remove them if you can’t, and note that you did.</p><p><strong>2. A rare truth.</strong> A value that’s extreme but genuine. A record-breaking sprint. A legitimate £4 million mortgage on a London house. A promotion day when website traffic jumps tenfold. Deleting these makes your data look neater and your model less prepared for the real world, which contains such cases.</p><p><strong>3. A signal.</strong> A value that’s unusual <em>because</em> something important is happening. The fraudulent transaction, the failing machine part, the cyber attack sending data to an unknown server, the abnormal test result that reveals a rare disease. Here the outlier isn’t a problem with the data. It’s the reason you’re analysing the data at all.</p><p>The trouble is that all three look exactly the same to a statistical rule: a point a long way from the rest.</p><h3>Why the usual rules can’t tell them apart</h3><p>The two standard tools are the z-score rule (flag anything more than three standard deviations from the mean) and the interquartile range rule (flag anything far outside the middle half of the data). Both are useful. Both measure <strong>distance</strong>, not <strong>meaning</strong>, and both have blind spots worth knowing.</p><p><strong>Big outliers can hide other outliers.</strong> The z-score uses the mean and standard deviation, and extreme values inflate both. In one classic exercise, 50 people guess an elephant’s weight. One joker writes 69,420 lbs. That single value stretches the standard deviation so much that a guess of 29,232 lbs, about three times the typical answer, scores a z of only around 2 and no longer looks extreme. The yardstick bends towards the thing it’s measuring.</p><p><strong>Rules can miss the obviously impossible.</strong> In the same data, the interquartile range rule correctly flags the two huge guesses. But its lower cut-off works out as a negative number, so guesses of 23 lbs and 320 lbs sail straight through. No statistical rule knows that an adult elephant can’t weigh as much as a Labrador. A person does.</p><p><strong>Unusual depends on context.</strong> A £5,000 card payment is routine for one customer and wildly out of character for another. A rule applied across the whole population compares each value to everyone. Fraud often only shows up when you compare a value to <em>that customer’s own</em> history, <em>that branch’s</em> usual volumes, or <em>that time of day</em>.</p><p>Statistical rules are excellent at producing a list of suspects. They can’t decide which suspects are guilty.</p><h3>Four questions before you delete anything</h3><p>When a value gets flagged, work through these in order.</p><ol><li><strong>Is it possible?</strong> Check it against the real world, not the data. Negative ages, 812 cm humans and 23 lb elephants are impossible, and impossible values are errors. This is where domain knowledge beats every formula.</li><li><strong>Can I explain it?</strong> Look at the rest of the record and its context. A huge transaction on Black Friday, a spike on a promotion day, a large mortgage in a high-value postcode: plausible explanations point to a rare truth.</li><li><strong>Is it related to what I’m trying to predict?</strong> If you’re modelling fraud, failures or disease, unusual values near the outcome are likely to be signal. Check whether flagged rows are more common among your positive cases. If they are, deleting them would gut your model.</li><li><strong>What does each choice cost?</strong> Keeping a genuine error can distort averages and model coefficients. Deleting a real signal can blind the model to the very cases that matter most. Which mistake would hurt more here?</li></ol><p>If you can’t answer the first two, the honest label is “unexplained”, and unexplained values deserve investigation, not deletion.</p><h3>Better options than delete</h3><p>Deleting is only one of several choices, and usually not the best.</p><ul><li><strong>Fix it.</strong> If a value is clearly an error and the true value is recoverable, correct it: a missing decimal point, a unit mix-up between pounds and pence.</li><li><strong>Flag it.</strong> Add a column marking the row as unusual and keep it. The model can then learn whether “unusual” itself matters, which in fraud it often does.</li><li><strong>Soften it.</strong> For genuine extremes that would otherwise dominate, transform rather than remove. A log scale lets a tenfold traffic spike still count without overwhelming everything else. Capping at a high percentile, known as winsorising, keeps the row but limits its pull.</li><li><strong>Model it separately.</strong> Sometimes the extremes behave differently enough to deserve their own treatment: very large loans, business accounts, overseas transactions.</li><li><strong>Never touch the raw data.</strong> Clean in a copy, keep the original, and record every rule you applied and how many rows it affected. Someone, possibly you in six months, will need to know.</li></ul><p>And always compare your results with and without the outliers. If a conclusion flips depending on a handful of rows, that fragility is itself an important finding.</p><h3>The twist: good fraud doesn’t look like an outlier</h3><p>There’s one more lesson, and it cuts the other way. The fraud that’s easy to spot, the enormous transaction or the obviously fake income, is also the fraud that basic rules already catch. Experienced fraudsters know this. They keep amounts just under review thresholds, spread activity across many accounts, and build up a history of normal behaviour before they strike. They work hard to look <em>ordinary</em>.</p><p>So outlier detection is a powerful first line of defence, not the whole answer. The strongest fraud systems combine it with models trained on confirmed past cases, comparisons against each customer’s own baseline, and relationships between records, such as shared addresses, devices or employers. And they treat today’s outliers as tomorrow’s training data, which only works if nobody deleted them.</p><h3>The takeaway</h3><p>An outlier is a question, not an answer. Statistics can tell you a value is far from the rest. Only context can tell you whether it’s a typo, a rare truth, or the very thing you’re looking for.</p><p>Before you clean your data, ask what each strange value is trying to tell you. In fraud detection, and in a surprising number of other places, the most valuable rows are the ones that look least like the others.</p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a89a803a097" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/dont-delete-your-outliers-yet-in-fraud-they-re-the-whole-point-6a89a803a097">Don’t delete your outliers yet. In fraud, they’re the whole point.</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The data filter that hid the Challenger disaster]]></title>
            <link>https://medium.com/the-learning-rate/the-data-filter-that-hid-the-challenger-disaster-e025fcf999b4?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e025fcf999b4</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[selection-bias]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 22:56:49 GMT</pubDate>
            <atom:updated>2026-11-01T22:56:49.865Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why deleting the “boring” rows can delete your signal</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwmkryfvsAHYbifh3TdWog.png" /></figure><p>On the night of 27 January 1986, engineers and managers held a tense teleconference about whether the space shuttle Challenger should launch the next morning. The forecast was unusually cold. The question on the table was simple: does cold weather damage the rubber O-ring seals in the rocket boosters?</p><p>They looked at the data from previous flights and couldn’t make a convincing case that it did. Challenger launched at about 36°F, the coldest launch ever attempted. Seventy-three seconds later, an O-ring seal failed and the shuttle broke apart, killing all seven crew members.</p><p>The data that would have shown the danger existed. It just wasn’t in the picture people were looking at, because of a filter that seemed perfectly sensible at the time. That filter is one of the most common mistakes in data work today.</p><h3>The picture that showed nothing</h3><p>Before Challenger, there had been 23 shuttle flights with recoverable boosters. On some of them, inspectors found O-rings showing signs of thermal distress: erosion or hot gas getting past the seal.</p><p>The natural question was whether temperature made damage more likely. And the natural first move, the one almost anyone would make, was to look at the flights where damage had actually happened. Why study flights where nothing went wrong?</p><p>There were seven of them. Plot their launch temperatures against the number of damaged O-rings and you get something like this:</p><ul><li>Damage at 53°F, the coldest launch on record.</li><li>But also damage at 70°F, and at 75°F, one of the warmest.</li><li>The worst cases, two damaged O-rings, appear at both 53°F and 75°F.</li></ul><p>There’s no pattern. Damage happens in the cold and in the warmth alike. The correlation between temperature and damage across those seven flights is essentially zero. Looking at that picture, the honest conclusion is that temperature doesn’t seem to matter. That was a large part of why the argument against launching didn’t carry the room.</p><h3>The rows that were missing</h3><p>Now put back the other 16 flights, the ones with no O-ring damage at all. They’re the “boring” rows, the ones nobody thought worth plotting. Every single one of them launched at 66°F or warmer.</p><p>The picture changes completely:</p><ul><li><strong>Every flight launched below 66°F had O-ring damage.</strong> All four of them.</li><li><strong>Most flights launched above 66°F had none.</strong> Damage in the warm range was the exception.</li></ul><p>Using the widely published data for those 23 flights, the correlation between temperature and damage jumps from about <strong>0</strong> to about <strong>−0.51</strong>: a clear, moderate negative relationship. Colder launches, more damage. And Challenger was about to launch at 36°F, seventeen degrees colder than the coldest previous flight, far outside anything the data had ever covered.</p><p>The Rogers Commission, which investigated the accident, made this point in its report: a careful analysis of the full flight history would have revealed the link between low temperature and O-ring damage.</p><p>Nothing was wrong with the data. Nothing was wrong with the arithmetic. The problem was a filter. By keeping only the flights where damage happened, the analysis removed exactly the rows that showed what “safe” looked like, and without them there was nothing to compare against.</p><h3>The same filter is everywhere</h3><p>The mistake has a name: <strong>selecting on the outcome</strong>. If you only study cases where the thing happened, you can’t see what makes it happen, because you’ve thrown away every case where it didn’t. A pattern needs contrast, and the contrast lives in the rows you deleted.</p><p>Once you know to look for it, it turns up constantly:</p><p><strong>Fraud investigations.</strong> A team analyses confirmed fraud cases to find warning signs. Brokered applications show up often, so “brokered” looks like a red flag. But if most applications come through brokers anyway, that pattern means nothing. Only a comparison with genuine applications can say whether a feature actually separates the two.</p><p><strong>Churn analysis.</strong> Studying only customers who left, to learn why customers leave, has the same blind spot. Customers who stayed may share every “churn reason” you find.</p><p><strong>Survivorship.</strong> In the Second World War, statisticians studying returning bombers noticed bullet holes clustered on the wings and fuselage. The famous insight, credited to Abraham Wald, was to armour where the holes <em>weren’t</em>: planes hit in those places never made it back to be counted. The missing rows were the most informative ones.</p><p><strong>Data pipelines.</strong> This is where the mistake hides best, because it looks like hygiene rather than analysis. A WHERE status = &#39;failed&#39; clause in an incident table. A dashboard that drops rows with zero values to keep charts tidy. An extract that keeps only transactions that were investigated. A deduplication step that quietly keeps one record per customer. Each one is reasonable on its own, and each one can remove the comparison group that analysts and models downstream depend on. The people using the data usually can’t see what was filtered out, so they can’t know what they’re missing.</p><h3>Five habits that catch it</h3><p>None of these need advanced statistics. They need the discipline to ask an awkward question at the moment a filter looks obviously right.</p><ol><li><strong>Ask what the filter removed, not just what it kept.</strong> Before any analysis, count the rows on both sides of every filter. If you kept 7 rows out of 23, the 16 you dropped deserve a look before you trust anything the 7 say.</li><li><strong>Plot the whole population first.</strong> Look at the full data before slicing it. Challenger’s danger is obvious on a chart of all 23 flights and invisible on a chart of 7.</li><li><strong>Keep the zeros.</strong> “Nothing happened” is data. Flights without damage, customers who didn’t leave, applications that weren’t fraud: these are your comparison group, not clutter.</li><li><strong>Compare, don’t just describe.</strong> A feature that appears in 80% of bad outcomes means nothing until you know how often it appears in good ones.</li><li><strong>Check whether you’re outside the data.</strong> Even with all 23 flights, a launch at 36°F was far colder than any previous flight. Any model, however good, is guessing when it’s asked about conditions it has never seen. When a decision depends on a prediction outside the range of your data, say so loudly.</li></ol><p>For anyone building data pipelines there’s a sixth: <strong>document every filter where the people downstream will see it.</strong> A one-line note saying “this table only includes investigated cases” can be the difference between a sound analysis and a confident wrong one.</p><h3>The quiet power of a WHERE clause</h3><p>It’s tempting to see Challenger as a story about pressure, deadlines and organisational failure, and it was all of those. But inside it sits a lesson for anyone who works with data: the most dangerous step in an analysis is often not the model or the maths. It’s the innocent-looking decision about which rows to include.</p><p>Filtering feels like tidying up. Removing the “nothing happened” cases feels like focusing on what matters. Sometimes it is. But a pattern only exists in contrast, and the contrast often lives in exactly the rows that look least interesting.</p><p>So the next time you write a filter, or receive a data set someone else has filtered, ask one question before anything else: <em>what would this look like if I put the boring rows back?</em></p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e025fcf999b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/the-data-filter-that-hid-the-challenger-disaster-e025fcf999b4">The data filter that hid the Challenger disaster</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your new model isn’t better. It just got lucky.]]></title>
            <link>https://medium.com/the-learning-rate/your-new-model-isnt-better-it-just-got-lucky-59939b46f40e?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/59939b46f40e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 21:13:55 GMT</pubDate>
            <atom:updated>2026-10-25T21:13:55.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ckO5gw2JzETcFGklrbQV-A.png" /></figure><p>You’ve spent two weeks on a new model. New features, better tuning, a cleaner pipeline. You run it on the test set: <strong>91% accuracy</strong>. The old model scored 90%.</p><p>It’s tempting to call it a win, write the update and move on.</p><p>But there’s a question almost nobody asks at that moment: if the two models were actually equally good, how often would one beat the other by a point just by chance? With a typical test set of 1,000 rows, the honest answer is “quite often”. Every model score is a random number, and a lot of celebrated improvements are just luck.</p><h3>A test score is a sample, not a fact</h3><p>Your test set is a sample of the cases your model will meet in the real world. Draw a different 1,000 rows and the same model would score a little higher or a little lower. Accuracy on a test set is really an average, the share of rows predicted correctly, and like every average it wobbles from sample to sample.</p><p>The central limit theorem tells you how much. The spread of an average, called its <strong>standard error</strong>, is the spread of the individual values divided by the square root of the sample size. For accuracy, that works out as:</p><blockquote><em>standard error ≈ √(accuracy × (1 − accuracy) ÷ number of test rows)</em></blockquote><p>At 90% accuracy on 1,000 rows: √(0.9 × 0.1 ÷ 1,000) ≈ <strong>0.95 percentage points</strong>.</p><p>So a model whose “true” accuracy is 90% will typically land anywhere from about 88% to 92% on a test set of that size. A one-point gap sits comfortably inside that wobble. In fact, if two equally good models were each scored on their own random test set of 1,000 rows, one would beat the other by a point or more almost half the time.</p><h3>More data helps, but slower than you’d hope</h3><p>The square root in that formula is the catch. Doubling your test set doesn’t halve the uncertainty; it shrinks it by only about 30%. To halve it, you need <strong>four times</strong> the data.</p><p>At 90% accuracy, the wobble looks like this:</p><ul><li><strong>1,000 test rows:</strong> about ±1 point</li><li><strong>4,000 test rows:</strong> about ±0.5 points</li><li><strong>40,000 test rows:</strong> about ±0.15 points</li></ul><p>This has a practical consequence before you train anything. If the improvement you’re hoping for is half a point, a test set of 1,000 rows simply can’t tell you whether you’ve achieved it. Decide how small a difference matters, and make sure your evaluation is big enough to see it.</p><p>The same logic applies to any rate you report: conversion, churn, default, fraud. A shift that looks meaningful on a small segment may be nothing more than the natural wobble of a small sample.</p><h3>Compare models the smart way</h3><p>You don’t always need a bigger test set. Often you need a better comparison.</p><p><strong>Compare on the same rows, case by case.</strong> Two overall scores hide what’s happening. Line the models up row by row and count the cases where they disagree. If the new model fixes 40 cases the old one got wrong but breaks 30 it got right, the net gain is 10 cases, and the question becomes whether 40 against 30 could easily happen by chance. A paired comparison like this is much more sensitive than comparing two headline numbers.</p><p><strong>Re-split and re-score.</strong> Train and evaluate both models across several different splits of the data, as cross-validation does. If the new model wins on almost every split, that’s convincing. If it wins on some and loses on others, the “improvement” was the luck of one particular split.</p><p><strong>Simulate the noise.</strong> Resample your test set with replacement a few thousand times and recompute the score each time. This bootstrap is a Monte Carlo simulation of what other test sets might have looked like, and the spread of results shows the uncertainty directly, without any formula.</p><p>None of these is complicated. What makes them rare is that they take an extra step at the exact moment everyone wants to declare victory.</p><h3>Even a real gain might not be worth it</h3><p>Suppose the checks come back and the improvement is genuine. That still doesn’t settle whether to ship it.</p><p><strong>Where did the gain land?</strong> One extra point of accuracy can come entirely from the easy majority class. In fraud detection, a model that’s slightly better at confirming genuine applications but no better at catching fraud has improved the number nobody cares about. Look at the cases that matter, not just the headline.</p><p><strong>What does it cost?</strong> A more complex model can be slower, harder to explain to a regulator or a customer, and harder to monitor. A small gain may not justify any of that.</p><p><strong>Will it last?</strong> Complex models that squeeze out the last fraction of a point on historical data are often the first to degrade when behaviour shifts. A simpler model that’s slightly worse today can be more reliable over a year.</p><p>The right question isn’t “is 91 bigger than 90?” It’s “is this improvement real, does it land where it matters, and is it worth what it costs?”</p><h3>Five questions before you declare a winner</h3><ol><li><strong>How big is the gap compared with the noise?</strong> At 90% accuracy, 1,000 test rows wobble by about a point.</li><li><strong>Did both models face the same rows?</strong> Compare them case by case, not just by headline score.</li><li><strong>Does the gain survive a re-split?</strong> One lucky split proves nothing.</li><li><strong>Is the gain in the cases that matter?</strong> Rare classes, high-value customers, the errors that cost money.</li><li><strong>Is it worth the cost?</strong> Complexity, explainability, monitoring and how long it will last.</li></ol><p>AI tools will train you a new model and report its accuracy to two decimal places in minutes. What they won’t tell you is whether the difference is real. Treating every score as a random number, with its own uncertainty, is one of the simplest habits that separates careful data scientists from lucky ones.</p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59939b46f40e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/your-new-model-isnt-better-it-just-got-lucky-59939b46f40e">Your new model isn’t better. It just got lucky.</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A 95% accurate fraud model. 96% of its alerts are wrong.]]></title>
            <link>https://medium.com/the-learning-rate/a-95-accurate-fraud-model-96-of-its-alerts-are-wrong-3c1749f67e26?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c1749f67e26</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[probability]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 21:09:29 GMT</pubDate>
            <atom:updated>2026-10-25T21:09:29.905Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OeJHpRtv2ujcdprbcU7L_g.png" /></figure><p>Imagine a fraud model that catches 95% of fraudulent mortgage applications and correctly clears 95% of genuine ones. By almost any measure, that’s an excellent model.</p><p>Now look at what happens when it raises an alert. Out of every 25 applications it flags, only one is actually fraud. The other 24 are honest people being treated as suspects.</p><p>Both statements are true at the same time. The model isn’t broken, and nobody has made an arithmetic mistake. It’s the base-rate trap, one of the most common and costly reasoning errors in data work. It catches data scientists, doctors, fraud teams and executives alike, and once you’ve seen it, you’ll notice it everywhere.</p><h3>The arithmetic, in plain counts</h3><p>Forget formulas for a moment and count people instead. Suppose fraud makes up 0.2% of applications, which is realistic for many lenders, and we score 100,000 of them.</p><ul><li><strong>200</strong> applications are fraudulent. The model catches 95% of them: <strong>190 true alerts</strong>.</li><li><strong>99,800</strong> applications are genuine. The model wrongly flags 5% of them: <strong>4,990 false alarms</strong>.</li></ul><p>That’s 5,180 alerts in total, and only 190 are fraud. <strong>About 3.7%.</strong> Put the other way round, roughly 96% of the alerts are wrong.</p><p>Nothing about the model changed between the first paragraph and this one. What changed is the question. “How often does the model catch fraud?” has the answer 95%. “When the model raises an alert, how often is it fraud?” has the answer 3.7%. Those are different questions, and the gap between them is entirely down to how rare fraud is.</p><h3>Why “accurate” hides the problem</h3><p>In probability language, the model’s headline figure is P(alert | fraud): the chance of an alert, given that an application is fraudulent. What the fraud team actually lives with is P(fraud | alert): the chance of fraud, given that an alert has fired. Swapping the two is so natural that most people don’t notice they’ve done it.</p><p>Bayes’ rule is the bridge between them. In words:</p><blockquote><em>The chance an alert is real = how good the model is at spotting fraud × how common fraud was to begin with ÷ how often alerts fire overall.</em></blockquote><p>The middle term, the <strong>base rate</strong>, is the one people forget. When fraud is rare, the 5% error rate is applied to a huge pool of genuine applicants, and those false alarms swamp the small number of true catches. A better model helps, but less than you’d think. Even at 99% accuracy both ways, only about 17% of alerts would be fraud.</p><p>This is also why “accuracy” is such a misleading headline for rare events. A model that never flags anything at all would be 99.8% accurate here, and completely useless.</p><h3>Once you see it, it’s everywhere</h3><p>The same pattern appears wherever you hunt for something rare.</p><p><strong>Medical screening.</strong> A disease affects 1% of people and a test is 90% accurate both ways. Of everyone who tests positive, only about 8% are actually ill. The other 92% are healthy people who face anxiety and follow-up tests. That’s why screening programmes follow a positive result with a second, more specific test before anyone is diagnosed.</p><p><strong>Security and spam.</strong> Systems that scan millions of passengers, emails or logins for rare threats inevitably flag far more innocent cases than real ones. The question is never whether there are false alarms, but how many you can afford to investigate.</p><p><strong>Data pipeline alerts.</strong> If you run data quality checks or anomaly alerts on production pipelines, you’ve met this too. A check that fires on 2% of healthy runs, monitoring a failure that happens on 0.1% of runs, will produce mostly false alarms. Teams quickly learn to ignore it, and then miss the real incident when it comes. The base-rate trap doesn’t just waste time; it erodes trust in the alert itself.</p><h3>What to do about it</h3><p>The base rate can’t be engineered away. It’s a fact about the world. But it should change how you design, report and act on a model.</p><p><strong>Treat alerts as evidence, not verdicts.</strong> A 3.7% hit rate is a strong reason to look closer and a terrible reason to decline someone automatically. Alerts belong in a review queue, with humans making the final call.</p><p><strong>Stack the evidence.</strong> Bayes’ rule works in sequence: today’s posterior becomes tomorrow’s prior. An alert lifts the chance of fraud from 0.2% to 3.7%. If a second, independent check that’s just as accurate, such as document verification, also flags the case, the chance rises to around 42%. Layered checks are how screening programmes and good fraud teams turn weak signals into strong ones.</p><p><strong>Report the right numbers.</strong> Instead of accuracy, report what each audience needs. Of the real fraud cases, how many did we catch? Of the alerts we raised, how many were real? The first is recall, the second precision. Both should sit on the dashboard.</p><p><strong>Set the threshold with the costs in view.</strong> A missed fraud might cost tens of thousands of pounds; a false alarm costs review time and some customer goodwill. Where to draw the line depends on those costs and on how many cases the team can realistically review, not on a default cut-off of 0.5.</p><p><strong>Watch the base rate itself.</strong> If fraud doubles, the share of real alerts roughly doubles with it, even though the model hasn’t changed. A model’s usefulness depends on the world it’s deployed into.</p><h3>Four questions to ask of any alert system</h3><p>Next time someone presents a model, a test or an alert with an impressive accuracy figure, ask:</p><ol><li><strong>How rare is the thing we’re looking for?</strong> The rarer it is, the more false alarms will dominate.</li><li><strong>When it fires, how often is it right?</strong> That’s the number the people acting on alerts actually experience.</li><li><strong>What happens next after an alert?</strong> A review, a second check, or an automatic decision? The rarer the event, the less an alert should decide on its own.</li><li><strong>What does each kind of mistake cost?</strong> That decides where the threshold belongs.</li></ol><p>A model can be excellent and still be wrong most of the times it speaks up. That isn’t a flaw to hide. It’s a fact to design around, and the teams who understand it build systems that people trust.</p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c1749f67e26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/a-95-accurate-fraud-model-96-of-its-alerts-are-wrong-3c1749f67e26">A 95% accurate fraud model. 96% of its alerts are wrong.</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 probability interview questions for data science and ML, and what interviewers actually listen…]]></title>
            <link>https://medium.com/the-learning-rate/10-probability-interview-questions-for-data-science-and-ml-and-what-interviewers-actually-listen-9002e02e1cfe?source=rss-46ed6bbe72cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/9002e02e1cfe</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Nidhin]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 20:53:35 GMT</pubDate>
            <atom:updated>2026-10-25T20:53:35.402Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>10 probability interview questions for data science and ML, and what interviewers actually listen for</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oeM6XRTr6Y2HXx6wPx6oYQ.png" /></figure><p>Probability questions are where data science /ML interviews quietly sort candidates. Most people can recite Bayes’ theorem. Far fewer can use it on a messy, realistic problem, or notice when a tidy assumption like independence is doing something dangerous.</p><p>Interviewers know this. When they ask a probability question, they’re rarely checking whether you remember a formula. They want to see whether you can reason about uncertainty out loud, sanity-check a number and connect the maths to a real decision.</p><p>Here are ten probability questions that come up in data science and machine learning interviews. Each answer has three parts: what a strong answer covers, what the interviewer is really listening for, and the answer that quietly costs you the role.</p><h3>1. “What’s the difference between probability and statistics, and where does each show up in machine learning?”</h3><p><strong>A strong answer:</strong> They run in opposite directions. Probability starts with a known model and asks what data it will produce: given a fair coin, how likely are 7 heads in 10 flips? Statistics starts with observed data and asks which model produced it: given 60 heads in 100 flips, is the coin fair? Machine learning runs both in a loop. Training is statistics, working backwards from historical data to a model’s parameters. Prediction is probability, working forwards from the trained model to likely outcomes. Monitoring is statistics again, checking predictions against what actually happened.</p><p><strong>What they’re listening for:</strong> The direction of reasoning, model to data or data to model, and the link to a real ML workflow rather than textbook definitions.</p><p><strong>The answer that costs you:</strong> “Statistics is about data and probability is about chance.” True, but it shows no understanding of how they connect.</p><h3>2. “Is P(A | B) the same as P(B | A)?”</h3><p><strong>A strong answer:</strong> No, and confusing them is one of the most common reasoning errors there is. P(A | B) means “the probability of A, given that B has happened”: you zoom into only the cases where B is true. The chance that a fraudulent application has a blank employer field is not the same as the chance that an application with a blank employer field is fraudulent. The first might be 40%; the second might be 2%, because most blank fields come from genuine applicants. Bayes’ rule is exactly the bridge that converts one into the other, using the base rates.</p><p><strong>What they’re listening for:</strong> A concrete example where the two clearly differ, and an awareness that Bayes’ rule connects them.</p><p><strong>The answer that costs you:</strong> “They’re basically the same, just written differently.” This is the reasoning error behind many bad fraud and medical decisions.</p><h3>3. “A fraud model catches 95% of fraud and wrongly flags 5% of genuine applications. Fraud is 0.2% of applications. What share of the model’s alerts are actually fraud?”</h3><p><strong>A strong answer:</strong> Think in counts. Out of 100,000 applications, 200 are fraudulent and the model catches 190. Of the 99,800 genuine ones, it wrongly flags 5%, which is 4,990. So there are 190 + 4,990 = 5,180 alerts, and only 190 of them are fraud: about <strong>3.7%</strong>. In Bayes’ rule terms, P(fraud | alert) = 0.95 × 0.002 ÷ (0.95 × 0.002 + 0.05 × 0.998). The model is “95% accurate” yet roughly 96% of its alerts are false alarms, purely because fraud is rare. In practice that means alerts should go to a review queue rather than trigger automatic declines, and you’d report precision and recall, not accuracy.</p><p><strong>What they’re listening for:</strong> Getting to the right number, ideally by converting to counts, and then explaining what it means for the business. The second part is what separates strong candidates.</p><p><strong>The answer that costs you:</strong> “95%.” This is the base-rate fallacy, and interviewers ask this question precisely to see whether you fall into it.</p><h3>4. “What does it mean for two events to be independent? Give an example where assuming independence goes wrong.”</h3><p><strong>A strong answer:</strong> Two events are independent if knowing one tells you nothing about the other, which means P(A and B) = P(A) × P(B). Coin flips and die rolls are independent; drawing cards without replacement isn’t, because each card removed changes what’s left. Independence is a convenient assumption, and dangerous when it’s false. Mortgage defaults are a classic example. Two loan books can have the same 2% average default rate, but if defaults rise together in a downturn, the bad years are far worse than a model assuming independence would predict. Independence also sits inside common models: Naive Bayes assumes features are independent given the class, which is rarely true but often works well enough.</p><p><strong>What they’re listening for:</strong> The multiplication rule, a non-obvious example of dependence, and awareness of what the assumption costs when it fails, especially in tail risk.</p><p><strong>The answer that costs you:</strong> A textbook definition with only coin examples, and no sense of where the assumption breaks in real data.</p><h3>5. “Explain the Monty Hall problem. Should you switch?”</h3><p><strong>A strong answer:</strong> Yes: switching wins two-thirds of the time. Your first pick is right only a third of the time. Monty knows where the car is and always opens a goat door, so his reveal doesn’t change the chance that your first pick was right; it’s still one third. The remaining two thirds must sit on the one door he left closed. Formally, with Bayes’ rule: if you pick door 1 and Monty opens door 2, the probability he does that is ½ if the car is behind door 1, 0 if it’s behind door 2 and 1 if it’s behind door 3. Weighting those by the equal priors gives ⅓ for sticking and ⅔ for switching. If I doubted the reasoning, I’d simulate 10,000 games, which takes a few lines of Python.</p><p><strong>What they’re listening for:</strong> The key insight that Monty’s choice isn’t random, because it depends on where the car is. Offering to verify by simulation is a strong signal.</p><p><strong>The answer that costs you:</strong> “It’s 50/50 because there are two doors left.” This ignores the information in Monty’s choice.</p><h3>6. “What is the binomial distribution, and when would you use it in practice?”</h3><p><strong>A strong answer:</strong> It models the number of successes in a fixed number of independent trials, each with the same success probability p. P(k successes in n trials) = C(n, k) × pᵏ × (1 − p)ⁿ⁻ᵏ, where C(n, k) counts the arrangements. Four conditions must hold: a fixed n, two outcomes per trial, constant p and independent trials. Practical uses: the number of customers who convert out of 1,000 visitors, the number of correct predictions out of a test set of 500, or the number of defaults in a loan book. The expected count is n × p and the standard deviation is √(np(1 − p)). I’d also check the conditions. If customers influence each other, or p varies a lot between customers, the binomial understates the spread.</p><p><strong>What they’re listening for:</strong> The four conditions, the formula with meaning attached, a realistic example, and when the model breaks.</p><p><strong>The answer that costs you:</strong> Reciting the formula without being able to say what the C(n, k) term is doing.</p><h3>7. “Explain the central limit theorem and why it matters in practice.”</h3><p><strong>A strong answer:</strong> If you take many samples of size n from almost any population and compute each sample’s mean, those means form a distribution that gets closer to normal as n grows, even if the population itself is skewed or lumpy. That distribution is centred on the population mean μ, and its spread, the standard error, is σ ÷ √n. In practice it matters because almost every number we report is an average: a conversion rate, a model’s accuracy, an average balance. The CLT tells us how uncertain each one is, which is the basis of confidence intervals and A/B tests. The √n also tells you that four times the data only halves the uncertainty, which matters when deciding how much data to collect.</p><p><strong>What they’re listening for:</strong> That it’s about the distribution of sample means, not the data itself becoming normal; the σ ÷ √n formula; and a practical use.</p><p><strong>The answer that costs you:</strong> “The CLT says data becomes normally distributed when you have enough of it.” The data keeps its shape; it’s the averages that become normal.</p><h3>8. “Your new model scores 91% accuracy on a 1,000-row test set. The old one scored 90%. Is the new model better?”</h3><p><strong>A strong answer:</strong> Not necessarily. Accuracy on a test set is itself an estimate with sampling error. At around 90% accuracy on 1,000 rows, the standard error is √(0.9 × 0.1 ÷ 1,000), roughly 0.95 percentage points, so a 1-point gap is about the size of the noise. Before concluding anything, I’d compare the two models on the same rows, case by case, which is more sensitive than comparing two overall scores. I’d use cross-validation or a larger test set to see whether the gain holds up, and check whether the improvement lands where it matters: one point of accuracy is worth little if it comes from the easy majority class rather than the rare cases the business cares about. And I’d weigh it against the cost of switching, such as complexity, explainability and monitoring.</p><p><strong>What they’re listening for:</strong> Treating a metric as a random quantity with uncertainty, a rough calculation of that uncertainty, and a practical plan to find out more.</p><p><strong>The answer that costs you:</strong> “Yes, 91 is higher than 90.” This shows no sense that test scores vary by chance.</p><h3>9. “What is Monte Carlo simulation, and when would you use it instead of an exact formula?”</h3><p><strong>A strong answer:</strong> Monte Carlo means estimating something uncertain by running a random experiment many times and studying the results. Build a model of the process, make the uncertain parts random, run it thousands of times, then read off averages, extremes and probabilities. I’d use it when an exact answer is hard or impossible: when many uncertain inputs interact, when events are correlated, or when I care about the tails rather than the average. Simulating 10,000 years of a loan portfolio, for example, shows how bad a 1-in-100 year could be once defaults are linked to the economy. It’s also a great way to check a formula you’re unsure about. The trade-off is precision: the error shrinks like 1 ÷ √N, so 100 times more simulations buys only 10 times more accuracy.</p><p><strong>What they’re listening for:</strong> A clear four-step description, a realistic use case where formulas struggle, and awareness of the cost in accuracy and computing.</p><p><strong>The answer that costs you:</strong> “It’s a random simulation” with no sense of when or why you’d use it, or how many runs you’d need.</p><h3>10. “X and Y are independent with standard deviations 3 and 4. What’s the standard deviation of X + Y? Why does it matter?”</h3><p><strong>A strong answer:</strong> <strong>5</strong>, not 7. For independent variables, variances add, so 9 + 16 = 25 and the standard deviation is √25 = 5. Random highs and lows partly cancel out when you add independent things. It matters in several places. Diversified portfolios are less volatile than their parts. Ensemble models reduce variance by averaging partly independent predictions. Averaging n measurements shrinks the spread by √n, which is the central limit theorem again. Two related rules are worth knowing: adding a constant shifts the mean but leaves the spread alone, and multiplying by a constant scales the standard deviation by that constant. That’s why standardising, z = (x − μ) ÷ σ, gives every feature a mean of 0 and a standard deviation of 1, the basis of feature scaling. If X and Y are correlated, though, the simple rule breaks: positive correlation makes the sum more variable.</p><p><strong>What they’re listening for:</strong> The correct answer with the reason (variances add), a real-world consequence, and the independence caveat.</p><p><strong>The answer that costs you:</strong> “7.” Adding standard deviations is one of the most common slips, and an interviewer will notice.</p><h3>The pattern behind every strong answer</h3><p>Look back across the ten and the same habits keep appearing. A strong probability answer:</p><ul><li><strong>Converts to counts when it can.</strong> “200 frauds in 100,000 applications” is far easier to reason about, and to explain, than a chain of conditional probabilities.</li><li><strong>Questions the assumptions.</strong> Independence, constant probability and normality are convenient, and the strongest candidates say what happens when they fail.</li><li><strong>Treats every number as uncertain.</strong> A test score, a conversion rate or an average comes with noise, and a good answer sizes that noise before drawing conclusions.</li><li><strong>Offers to check by simulation.</strong> A few lines of code that confirm the reasoning show both rigour and practicality.</li><li><strong>Ends with a decision.</strong> The maths matters because of what it changes: a review queue, a capital buffer, whether to ship a model.</li></ul><p>The formulas are easy to look up. What interviewers are really testing is whether you can reason clearly when you can’t be sure.</p><p><em>I’m learning machine learning and AI and writing up what I learn in The Learning Rate.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9002e02e1cfe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-learning-rate/10-probability-interview-questions-for-data-science-and-ml-and-what-interviewers-actually-listen-9002e02e1cfe">10 probability interview questions for data science and ML, and what interviewers actually listen…</a> was originally published in <a href="https://medium.com/the-learning-rate">The Learning Rate</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>