<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hamza Ahmed on Medium]]></title>
        <description><![CDATA[Stories by Hamza Ahmed on Medium]]></description>
        <link>https://medium.com/@alihamzabham?source=rss-d86d32b6eb9a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*AqudZ6ElJcqYA341CYYfxA.jpeg</url>
            <title>Stories by Hamza Ahmed on Medium</title>
            <link>https://medium.com/@alihamzabham?source=rss-d86d32b6eb9a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:39:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alihamzabham/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Normalizing Numerical Features: Linear, Z-Score, Log Scaling, and Clipping]]></title>
            <link>https://medium.com/h7w/normalizing-numerical-features-linear-z-score-log-scaling-and-clipping-2abe91b32392?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2abe91b32392</guid>
            <category><![CDATA[numerical-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Sun, 20 Sep 2026 14:14:06 GMT</pubDate>
            <atom:updated>2026-09-25T07:56:08.130Z</atom:updated>
            <content:encoded><![CDATA[<p>The goal of <strong>normalization</strong> is to transform features so they sit on a similar scale.</p><p>Imagine two features in the same dataset:</p><ul><li>Feature X spans the range 154 to 24,917,482.</li><li>Feature Y spans the range 5 to 22.</li></ul><p>These two features are on wildly different scales. If you feed them into a model as-is, the model has no inherent sense that a difference of a million in X isn’t comparable to a difference of one in Y. Normalization fixes this by transforming features so they span a similar range, often 0 to 1.</p><p>The benefits of normalization include:</p><ul><li>Models <strong>converge faster </strong>during training.</li><li>Models produce <strong>better predictions</strong>.</li><li>Models can <strong>learn appropriate weights </strong>for each feature, rather than being thrown off by scale differences.</li></ul><p>As a rule of thumb: normalize when you have numeric features covering very different ranges (age vs. income, for example), and also normalize any single feature that spans a wide range on its own (like city population).</p><p>Consider a dataset having these two features:</p><ul><li>Feature A’s lowest value is -0.5 and highest is +0.5.</li><li>Feature B’s lowest value is -5.0 and highest is +5.0.</li></ul><p>Feature B’s range is 10 times wider than Feature $A$’s range. Therefore:</p><ul><li>At the start of training, the model assumes that Feature B is ten times more “important” than Feature A.</li><li>Training will take longer than it should.</li><li>The resulting model may be suboptimal.</li></ul><p>One important note: <em>whatever normalization you apply during training, you must apply the exact same transformation at prediction time.</em> If you normalize a feature during training but forget to do so before inference, your model will receive inputs it’s never seen the likes of, and predictions will suffer.</p><p>The normalization techniques worth knowing are:</p><ul><li>linear scaling</li><li>Z-score scaling</li><li>log scaling</li></ul><p>Additionally, we will cover clipping which is not a true normalization technique but does constrain features into a specified range.</p><h4>Linear Scaling</h4><p>Linear scaling converts floating-point values from their natural range into a standard range, usually 0 to 1 or -1 to +1. The formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/0*GqcpOIwQ-O6AcPWD.png" /></figure><ul><li>x’ is the scaled value.</li><li>x is the original value.</li><li>x_min is the lowest value in the dataset of this feature.</li><li>x_max is the highest value in the dataset of this feature.</li></ul><p>Linear scaling works best when:</p><ul><li>The lower and upper bounds of the data don’t change over time.</li><li>The feature contains few outliers.</li><li>The feature is approximately uniformly distributed across its range.</li></ul><p>If any of those conditions aren’t met, say, one giant outlier defines your x_max, linear scaling can squash the rest of your data into a tiny sliver of the 0–1 range, which defeats the purpose.</p><p>Suppose human <em>age</em> is a feature. Linear scaling is a good normalization technique for <em>age </em>because <em>age</em> contains a relatively small percentage of outliers.</p><h4>Z-score scaling</h4><p>A <strong>Z-score</strong> represents how many standard deviations a value is from the mean. Z-score scaling stores that Z-score in the feature vector instead of the raw value.</p><p>For example, a value that is 2 standard deviations <em>greater</em> than the mean has a Z-score of +2.0. A value that is 1.5 standard deviations <em>less</em> than the mean has a Z-score of -1.5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/91/0*dVATOIaG26OXrNIG.png" /></figure><ul><li>x’ is the Z-score.</li><li>x is the raw value; that is, is the value you are normalizing.</li><li>μ is the mean.</li><li>σ is the standard deviation.</li></ul><p>Z-score scaling is a strong choice when your data follows a normal distribution, or something reasonably close to one. Unlike linear scaling, it doesn’t depend on fixed min/max bounds, so it handles moderate outliers more gracefully.</p><h4>Log scaling</h4><p>Log scaling computes the logarithm of the raw value. In practice, usually the natural logarithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/0*nPQ-sy0CyhCmqYyS.png" /></figure><p>This technique shines when your data follows a <strong>power law distribution, </strong>that means:</p><ul><li>Low values of <em>X</em> correspond to very high values of <em>Y</em> (or vice versa).</li><li>As <em>X</em> increases, <em>Y</em> drops off quickly.</li></ul><p>Log scaling compresses that skew into a more manageable, evenly spread-out distribution, which helps the model learn patterns it would otherwise struggle to see in the raw, heavily skewed values.</p><h4>Clipping</h4><p>Clipping is less about rescaling and more about damage control. It minimizes the influence of extreme outliers by capping (reducing) their value to a specified maximum and/or minimum.</p><p>Rather than letting one extreme data point distort your scaling or drag your model’s learned weights around, clipping simply says: “past this point, treat every value the same.” It’s often used <em>alongside</em> one of the other three techniques. For example, clipping outliers before applying linear scaling.</p><h4>Quick Reference</h4><p>Following table shows which technique to use for normalization according to your data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v2d77fqEuxqcrn_ea2WL2Q.png" /></figure><h4>Conclusion</h4><p>Normalization isn’t a single one-size-fits-all step, it’s a decision point. The right technique depends on the shape of your data’s distribution, and getting it wrong (or skipping it entirely) can slow down training or actively hurt model performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2abe91b32392" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/normalizing-numerical-features-linear-z-score-log-scaling-and-clipping-2abe91b32392">Normalizing Numerical Features: Linear, Z-Score, Log Scaling, and Clipping</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Numerical Data & Feature Vectors]]></title>
            <link>https://medium.com/h7w/introduction-to-numerical-data-feature-vectors-82cd81e866a9?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/82cd81e866a9</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[numerical-data]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 15:38:02 GMT</pubDate>
            <atom:updated>2026-09-15T15:38:02.824Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’re getting started with machine learning, one of the first things you need to internalize is that <em>not everything that looks like a number actually behaves like one.</em> That distinction is the entire starting point for numerical data and feature engineering.</p><h4>What counts as “Numerical Data”</h4><p>Numerical data means integers or floating-point values that behave like numbers. They’re additive, countable, and ordered. Think temperature, weight, or the number of students in a class. If it makes sense to add two values together, average them, or say one is “bigger” than the other, it’s numerical.</p><p>In contrast, some things <em>look</em> numerical but aren’t. Postal codes are a classic example. A ZIP code is a five- or nine-digit number, but adding two ZIP codes together or averaging them produces nonsense. These numbers represent categories and are considered <em>Categorical Data</em>.</p><h4>How a Model Actually Ingests Data: The Feature Vector</h4><p>Suppose the data contains five columns, of which 2 columns are being used as features for the model. Does the model just take the values of those features to process?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gUOzCdnFPx1YRaIQestnw.png" /><figcaption>Example Without Feature Vector</figcaption></figure><p>Not really. In fact, the model ingests an array of floating-point values called a <strong>feature vector</strong>. A feature vector consists of floating-point values in an array or vector that will be considered one example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxvX149kL1K9hYoULfq9hA.png" /><figcaption>Example With Raw Feature Vector</figcaption></figure><p>The catch is that feature vectors rarely use your dataset’s raw values as-is. Instead, you typically need to process those raw values into a representation the model can learn from more effectively. Realistically, a feature vector might look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3q3dYDmR2e0LOnJUk7csA.png" /><figcaption>Example with processed Feature Vector</figcaption></figure><p>Turning raw data into that trainable representation is called <strong>feature engineering</strong>, and it’s one of the most consequential parts of building a machine learning model. Two of the most common feature engineering techniques are:</p><ul><li><strong>Normalization</strong>: Converting numerical values into a standard range.</li><li><strong>Binning (bucketing)</strong>:<strong> </strong>Turning continuous numerical data into discrete categories or groups (bins).</li></ul><p>Every value that goes into a feature vector has to end up as a floating-point number. Since plenty of real-world features start out as strings or other non-numerical values, a large chunk of feature engineering work is simply about converting non-numerical data into something numerical the model can use.</p><h4>First Steps Before Building a Feature Vector</h4><p>Before you touch feature engineering, it’s worth spending time actually getting to know your data. The following steps are recommended:</p><p><strong>Visualize Your Data<br></strong>Graphs can reveal anomalies, trends, and patterns that are easy to miss in a table of numbers. Use <strong>histograms and box plots</strong> to understand distributions and spot outliers, <strong>scatter plots</strong> to examine relationships between variables, and <strong>bar or line charts</strong> to compare values or visualize trends over time. In Python, <strong>Matplotlib and Seaborn</strong> are good choices for most data exploration, while <strong>Plotly</strong> can be useful when interactive visualizations are needed.</p><p>It is worth creating visualizations throughout the data pipeline, not just at the beginning. Data cleaning and transformations can introduce new issues that may only become apparent when you visualize the data again.</p><p><strong>Statistically evaluate your data</strong><br>Basic summary statistics such as the <strong>mean, median, standard deviation, and quartiles</strong> give you a quick quantitative understanding of your data. They can help reveal the typical values, spread, and potential outliers. In practice, this often looks like:</p><pre>training_df = pd.read_csv(...)<br>training_df.describe()</pre><p><strong>Find outliers</strong><br>An <strong>outlier</strong> is a value that sits far from most other values in a feature or label. Outliers can quietly distort model training if they go unnoticed, so it’s worth actively checking for them rather than assuming your data is clean. A simple way to identify potential outliers is to use a <strong>box plot</strong> or the <strong>IQR (interquartile range)</strong> method. In pandas, you can quickly inspect a feature with:</p><pre>training_df[&quot;feature&quot;].plot.box()</pre><p>Not every outlier is a problem. First, check <strong>why it is an outlier</strong>. If it comes from a mistake, such as an incorrect value or faulty measurement, it should usually be removed. If it is a legitimate value, consider whether your model will need to make predictions for similar cases. If so, keep it. Otherwise, you can remove it or use techniques such as <strong>clipping </strong>(limits or caps numerical values) to limit its effect.</p><h4>Conclusion</h4><p>Before building a machine learning model, it is important to understand what your data actually represents. Not every value that looks like a number should be treated as numerical data. By exploring distributions, checking statistics, identifying outliers, and understanding how raw values become feature vectors, you can prepare data in a form that a model can learn from effectively. Good feature engineering starts with good data understanding, so taking the time to inspect your data before modeling can prevent many problems later in the pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82cd81e866a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/introduction-to-numerical-data-feature-vectors-82cd81e866a9">Introduction to Numerical Data &amp; Feature Vectors</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Classification: Thresholds, Metrics, ROC/AUC, and Bias]]></title>
            <link>https://medium.com/h7w/understanding-classification-thresholds-metrics-roc-auc-and-bias-c13fc2f85b0a?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c13fc2f85b0a</guid>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[performance-metrics]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 00:18:48 GMT</pubDate>
            <atom:updated>2026-09-11T00:18:48.268Z</atom:updated>
            <content:encoded><![CDATA[<p>Classification is the task of predicting which of a set of classes (categories) an example belongs to. When there are only two possible classes, it’s called <strong>binary classification</strong>, logistic regression is an example of a model that can be converted into a <em>binary classification.</em></p><h4>Threshold and the Confusion Matrix</h4><p>A logistic model doesn’t just output “yes” or “no”, it outputs a <em>probability</em>. To turn that probability into a decision, we set a <strong>classification threshold</strong> (also called a decision threshold). Any prediction above the threshold is assigned to the positive class (eg: spam); anything below is assigned to the negative class (eg: not spam).</p><p>A common default is 0.5, but the “right” threshold isn’t universal. It would not be a good idea if the cost of one type of wrong classification is greater than the other, or if the classes are imbalanced. So the suitable threshold depends on the dataset and the real-world cost of getting a prediction wrong. Missing a fraud case and flagging a legitimate transaction as fraud don’t cost the same, so the threshold should reflect that.</p><p>Once you have predictions and actual outcomes, there are four possible outcomes for each output from a binary classifier. if you lay out the ground truth as columns and the model’s prediction as rows, you can organize them into a <strong>confusion matrix</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FjHcsPHKRl_fEj_tUkR4bA.png" /></figure><ul><li>The total in each row gives all predicted positives (TP + FP) and all predicted negatives (FN + TN)</li><li>The total in each column gives all real positives (TP + FN) and all real negatives (FP + TN) regardless of model classification.</li></ul><p>If the number of actual positives is far from the number of actual negatives, the dataset is considered <strong>imbalanced. </strong>An imbalanced dataset is when one category vastly outnumbers another, such as a medical database containing hundreds of thousands of patient scans where the specific brain tumor you are looking for only appears in a handful of cases.</p><p><strong>Effect of moving the threshold:</strong> Raising the threshold makes the model more conservative about predicting “positive,” which typically <em>reduces</em> false positives but <em>increases</em> false negatives. Lowering the threshold does the opposite. Every metric derived from the confusion matrix shifts as you slide this threshold up or down.</p><h4>Accuracy, Precision, Recall, and F1</h4><p>The confusion matrix is used to calculate several metrics for evaluating models. Which one matters most depends on your model, your task, the cost of misclassifying each class, and whether your dataset is balanced or imbalanced. Often the threshold is tuned to optimize one of these metrics.</p><p><strong>Accuracy:<br></strong>The proportion of all classifications that were correct:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*xIRSZVwcih-5FChR.png" /></figure><p>A perfect model would have an accuracy of 1.0, or 100% i.e. zero false positives and zero false negatives.</p><p>On a balanced dataset, accuracy works well as a basic baseline because it accounts for all four outcomes of a confusion matrix. Since it provides a quick, high-level snapshot of overall model performance, it is the most common default metric for generic machine learning tasks.</p><p>Accuracy can be misleading on imbalanced datasets. A model that always predicts “not spam” on a dataset that’s 95% legitimate email will score 95% accuracy while being useless.</p><p><strong>Recall, or True Positve Rate (TPR):</strong><br>The proportion of actual positives that were correctly identified:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/0*VQU93hrdG8ERdoKX.png" /></figure><p>Recall measures a model’s ability to spot the “positives.” Also known as the <em>probability of detection,</em> it calculates the percentage of actual positive cases the model successfully finds. For example, in a spam filter, recall answers: <em>“What fraction of real spam emails did the model actually catch?”</em></p><p>On imbalanced datasets, recall is far more valuable than accuracy. In high-stakes scenarios like medical testing or fraud detection, a false negative can be catastrophic, missing a disease is much worse than triggering a false alarm. In these cases, you must prioritize recall to ensure no critical cases slip through.</p><p><strong>False Positive Rate (FPR):</strong><br>The proportion of actual negatives that were incorrectly flagged as positive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/0*e7Vh7UnEWPviYJ3u.png" /></figure><p>The False Positive Rate (FPR) measures a model’s false alarms, calculating the percentage of normal cases mistakenly flagged as positive. In a spam filter, FPR tells you how many legitimate emails were accidentally classified as spam. A perfect model scores an FPR of 0.0 (0%), meaning it never triggers a false alarm.</p><p><strong>Precision:<br></strong>The proportion of positive <em>predictions</em> that were actually correct:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/0*0XpZgZvg4VrpbpOt.png" /></figure><p>Precision measures the percentage of predicted positives that are actually correct. In a spam filter, it answers: <em>“Of all the emails flagged as spam, how many were actually spam?”</em> A perfect model scores a <em>1.0 (100% precision)</em> by triggering zero false alarms, though the metric becomes less reliable if the dataset contains only one or two actual positive examples.</p><p>In practice, precision and recall share an inverse relationship.</p><p><strong>F1 Score:<br></strong>The harmonic mean of precision and recall, useful when you want a single number that balances both:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/0*wO3dMAfFNScSFJoO.png" /></figure><h4>ROC and AUC</h4><p><strong>ROC curve (Receiver Operating Characteristic curve):<br></strong>Rather than evaluating a model at a single threshold, the <strong>ROC curve</strong> (Receiver Operating Characteristic curve) shows how a model performs across <em>every possible threshold</em>.</p><p>It’s built by calculating TPR and FPR at many threshold values, then plotting TPR (y-axis) against FPR (x-axis). A perfect model reaches a TPR of 1.0 and an FPR of 0.0 at some threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*IiHxYNHo6mfAqH5EM3rfjw.png" /><figcaption>ROC Curve</figcaption></figure><p><strong>AUC (Area Under the Curve):<br></strong>AUC condenses this curve into a single number: the probability that the model, given one randomly chosen positive example and one randomly chosen negative example, ranks the positive one higher.</p><ul><li>A perfect model has an AUC of 1.0.</li><li>Random guessing (a coin flip) has an AUC of 0.5.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ix_qvcKrg26cZDmDvNbQIg.png" /><figcaption>ROC and AUC for random and perfect classifier</figcaption></figure><p><strong>Using ROC and AUC to choose a model or threshold</strong><br>AUC is useful for comparing two different models, higher AUC generally means better performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*teyezcpyqehP48KczdG-yA.png" /><figcaption>ROC and AUC of two models</figcaption></figure><p>This comparison is most meaningful when the dataset is roughly balanced. On the ROC curve itself, the point closest to the top-left corner (0 FPR, 1 TPR) marks the range of best-performing thresholds. Ultimately, though, the threshold you pick should depend on which metric (precision, recall, etc.) matters most for your specific use case.</p><p>The best-performing thresholds for a model are found at the points on a ROC curve closest to the top-left corner at coordinates (0,1).</p><h4>Prediction Bias</h4><p>Prediction bias is a fast sanity check to catch problems early, before digging into deeper evaluation. It’s simply the difference between the mean of the model’s predictions and the mean of the ground-truth labels.</p><p>For example, if a dataset is genuinely 5% spam, a well-calibrated model should predict “spam” about 5% of the time. Zero bias means those two numbers match.</p><p>Prediction bias could be due to:</p><ul><li>Bias or noise in the training data</li><li>Overly strong regularization, which oversimplifies the model</li><li>Bugs somewhere in the training pipeline</li><li>A feature set that’s insufficient to capture the real signal</li></ul><h4>Multi-Class Classification</h4><p>Not every problem has just two classes. For classes A, B, and C, one common strategy is to decompose the problem into a series of binary classifications, for instance, first classify [A + B] vs. C, then separately classify A vs. B.</p><p>It’s also worth distinguishing this from <strong>multi-label classification</strong>, where class membership isn’t mutually exclusive and a single example can legitimately belong to more than one class at once.</p><h4>Conclusion</h4><p>Classification is more than simply predicting the correct class. A model’s performance depends on how its predictions are interpreted, which threshold is chosen, and which types of errors matter most for the problem.</p><p>The confusion matrix provides the foundation for understanding these errors, while metrics such as accuracy, precision, recall, F1 score, and false positive rate offer different perspectives on model performance. ROC curves and AUC extend this evaluation across different thresholds, making them useful for comparing models and understanding the trade-off between detecting positives and generating false alarms.</p><p>There is no single metric or threshold that is best for every classification problem. The right choice depends on the dataset, class balance, and real-world consequences of false positives and false negatives. Understanding these trade-offs is therefore just as important as building the classifier itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c13fc2f85b0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/understanding-classification-thresholds-metrics-roc-auc-and-bias-c13fc2f85b0a">Understanding Classification: Thresholds, Metrics, ROC/AUC, and Bias</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression Explained]]></title>
            <link>https://medium.com/h7w/logistic-regression-explained-ed7c6f0b299f?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed7c6f0b299f</guid>
            <category><![CDATA[log-loss]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regularization]]></category>
            <category><![CDATA[sigmoid-function]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 18:18:20 GMT</pubDate>
            <atom:updated>2026-09-06T18:18:20.315Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/h7w/linear-regression-explained-b6d94244876a">linear regression</a>, we’ve seen that how model makes continous numerical predictions such as amount of time to drive to destination but what if we need to answer like “is this email a spam?” or “will it rain today?”. This where a new type of regression known as <em>Logistic Regression</em> come in, which predicts the probability of an outcome.</p><h4>How Does it Calculate Probability?</h4><p>Logisitic regression calculates probability of an outcome which can be used in either of the two following ways:</p><ul><li>Return probability (percentage) as-is, if a model predicts if an email is spam to be 0.89 which means with 89% probability this email is a spam.</li><li>Convert to a binary category, such as <em>True</em> or <em>False </em>(<em>spam</em> or <em>not spam</em>)</li></ul><p>Here, we will be using the output as-is and not convert to a category.</p><h4>Sigmoid Function</h4><p>Sigmoid (means S-shaped) function is used in logistic regression to ensure the output is always between 0 and 1 i.e returning a probability. This is a standard logistic function, and has the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/0*wZbJssPRIojvALNB.png" /></figure><ul><li>f(x) is output of sigmoid function</li><li>e is Euler’s Number, a constant $\approx 2.71828 $</li><li>x is input to sigmoid function</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*qBSXMevJFD2au9p1ONLRxw.jpeg" /><figcaption>Sigmoid Function Graph</figcaption></figure><p>As input increases, the sigmoid function approaches to 1. Similarly, as input decreases, the sigmoid function approaches to 0.</p><h4>Using the sigmoid function on Linear Output</h4><p>Following is the linear regression model or linear output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/0*6dxnVyNG9hNIG8FC.png" /></figure><ul><li>z is the output of the linear equation</li><li>b is bias</li><li>The w values are the learned weights</li><li>The x are the feature values</li></ul><p>To get the logistic regression outcome, the $z$ value is passed to the sigmoid function, providing values between 0–1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*QFLC1vmekADtdGfd.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLC19XII5msl3JXd3PrsmQ.png" /></figure><p>The above figure shows a linear grpah (straight line) is converted to an S-shaped graph which always outputs between 0–1 i.e probability.</p><h4>Loss and Regularization</h4><p>Logistic regression applies:</p><ul><li><em>Log Loss</em> as loss funtion instead of squared loss (as in linear regression)</li><li><em>Regularization</em> to prevent overfitting</li></ul><p><strong>Log Loss</strong></p><p>In linear regression, Squared Loss works well because the rate of change of output is constant. Whereas, rate of change of output in logistic regression is not constant, the sigmoid function is S-shaped. Small change in input results in larger changes to output than input is large positive or negative number.</p><p>When using squared loss with a sigmoid function, outputs near 0 and 1 require higher computational precision, and therefore more memory, to accurately track the tiny differences in error.</p><p>Logistic regression uses <em>Log Loss, </em>it returns the logarithm of the magnitude of the change, rather than simply distance from data to prediction. Log loss is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/0*ridVN_UF4sCvi5TQ.png" /></figure><ul><li>N is the number of labeled examples in dataset.</li><li>i is the example in the dataset.</li><li>yi is the label of $ith$ example.</li><li>y’i is model’s prediction for the the $ith$ example.</li></ul><p><strong>Regularization</strong></p><p><em>Regularization</em> is a technique that penalizes model complexity during training. It is incredibly important for logistic regression models.</p><p>Without regularization, a logistic regression model with too many features will try too hard to fit the training data perfectly. Because of how the mathematics behind the model work, it will continuously push its loss toward 0. This creates an overly complex model that memorizes the training data but fails to predict new, real-world data accurately.</p><p>Most logistic regression models use one of two simple strategies to reduce complexity:</p><ul><li><strong>L2 Regularization: </strong>Adds a penalty based on the squared size of the weights to the loss function. This shrinks all weights proportionally toward zero, preventing any single feature from dominating the predictions.</li><li><strong>Early Stopping:</strong> halt training while loss is still decreasing, by limiting the number of training steps.</li></ul><h4>Conclusion</h4><p>Logistic regression extends linear regression by passing its output through the sigmoid function, converting any real-numbered result into a probability between 0 and 1. Because this S-shaped curve behaves differently than a straight line, standard squared loss doesn’t work well, log loss is used instead. And since logistic regression models can easily overfit by chasing zero loss, regularization (like L2, or simply limiting training steps) is essential to keep the model generalized to new data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed7c6f0b299f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/logistic-regression-explained-ed7c6f0b299f">Logistic Regression Explained</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Loss and Gradient Descent: How a Model Learns]]></title>
            <link>https://medium.com/h7w/understanding-loss-and-gradient-descent-how-a-model-learns-43e90ebb9c36?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/43e90ebb9c36</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[aritificial-intelligence]]></category>
            <category><![CDATA[ai-model]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 17:31:41 GMT</pubDate>
            <atom:updated>2026-09-07T19:16:49.334Z</atom:updated>
            <content:encoded><![CDATA[<p>A model could start with almost any values. Some values will produce predictions that are close to the actual values, while others will produce very poor predictions.</p><p>To find better values, we need a way to measure how wrong the model is and a way to improve it.</p><p>This is where <strong>loss</strong> and <strong>gradient descent</strong> come in.</p><h4>What is Loss?</h4><p><strong>Loss</strong> is a numerical metric that measures how far a model’s predictions are from the actual values. It calculates the distance between the model’s predictions and the actual labels.</p><p>A lower loss means the model’s predictions are generally closer to the actual values. A higher loss means the predictions are further away. The goal of training a model is to minimize the loss.</p><p>In the following image, the arrows show how far the model’s predictions are from the actual values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*q-aaN6_tQHVazK6E-9eNYQ.png" /><figcaption>Loss</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*5DbC78NYRkbiN1xK6Wl1Vg.png" /><figcaption>Errors Table</figcaption></figure><p>In Loss, we only care about the <em>distance, </em>and not the direction or sign. Thus, all methods for calculating loss remove the sign. Following are the two common ways to remove sign from value:</p><ul><li>Take the absolute value of the difference.</li><li>Square the difference.</li></ul><h4>Types of Loss</h4><p><strong>L1 loss: </strong>The sum of the absolute values of the difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*H6D4-7kNhlZ7Eo_H.png" /></figure><p><strong>Mean absolute error (MAE): </strong>The average of L1 losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/114/0*ipqSHJnF_gUrgC3U.png" /></figure><p><strong>L2 loss: </strong>The sum of the squared difference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/103/0*BvskLemtXYtKrua4.png" /></figure><p><strong>Mean squared error (MSE): </strong>The average of L2 losses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/0*FjzkHXO9W655iyWC.png" /></figure><p><strong>Root mean squared error (RMSE): </strong>The square root of the (MSE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/0*tEVl24E15ix9noot.png" /></figure><p>The difference between L1 loss and L2 loss (or between MAE and MSE) is squaring, which means:</p><ul><li><strong>L1 Loss or MAE</strong>: It is generally <strong>more robust to outliers </strong>because it uses the absolute value of the error. Since the error increases linearly, a very large error does not have an excessively large impact on the overall loss.</li><li><strong>L2 Loss or MSE</strong>: It is <strong>more sensitive to outliers </strong>because it squares the errors. This causes large errors to contribute much more to the loss than smaller ones. As a result, the model tends to give more importance to outliers and may adjust its predictions more strongly toward them.</li></ul><p>Now we can define the goal of training much more clearly, that is to find the model parameters (<em>weight </em>and <em>bias </em>in case of linear regression) that produce the <strong>lowest possible loss</strong> (whichever type of loss is chosen, depending on the dataset and how you want to handle predictions).</p><h4>What Is Gradient Descent?</h4><p>Gradient descent is a mathematical technique that iteratively finds the weights and bias that produce a model with loweest loss.</p><p>It starts with random weights and biases, usually initialized with small values near zero. The model then calculates the loss using the current weight and bias values and determines the direction in which they need to be adjusted to reduce the loss. It makes a small update to the weights and bias in that direction and then repeats the process. This continues until the model reaches a point where the loss can no longer be reduced significantly.</p><p>Let’s use a small house-price dataset to see exactly how gradient descent learns the weight and bias.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*NcjtqGBsuvomBeXml_s1Xw.png" /><figcaption>House Data</figcaption></figure><p>Our <a href="https://medium.com/blog/linear-regression-explained">linear regression</a> model is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/0*5TgZ5lRrvVmBKrQ5.png" /></figure><p>Let’s assume the model starts with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/0*e6f-weoE8p8EMUqG.png" /></figure><p>So the model is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/90/0*d8gh0aNvIBPcIxVb.png" /></figure><p>We’ll use <strong>Mean Squared Error (MSE)</strong> to measure how far our predictions are from the actual prices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/0*EFYinZQD5BqgIzNc.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/0*ob9hPrgCazpqkWye.png" /></figure><p>Now we need to determine <strong>which direction we should move the weight and bias</strong> to reduce the loss. This is where the gradient is useful. The gradient or slope of tangent of parameters can be calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/0*BxM1wEeO8w19iATk.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/0*OC9p6Nzsth1BB4dX.png" /></figure><p>and the calculated gradient values are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/133/0*y8CTlSaodOWDerwy.png" /></figure><p>Both gradients are negative.</p><p>This tells us that increasing the weight and bias will move us toward a region with lower loss.</p><p>Gradient descent updates the parameters using:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*OLxaOY0AdGvMkXNT.png" /></figure><p>Let’s use a learning rate of:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/90/0*a2tcrkaUmq0jamos.png" /></figure><p>updated weight and bias will be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*lCF29jMOO-IW7AzL.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*pWD2zps-CMwEzc8W.png" /></figure><p>with updated parameters, the model has now changed to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/0*QOJZvpBwXZxaUHGa.png" /></figure><p>The new <strong>MSE </strong>is approximately:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/0*XaDSfPkv9NG0I4Me.png" /></figure><p>The predictions are still far from the actual prices, but the model has started moving in the right direction. The loss has decreased.</p><p>We calculate the gradients again using the new weight and bias, update the parameters again, and calculate the loss again, and with each iteration the loss tend to decrease until convergence.</p><h4>Model convergence and loss curves</h4><p>While training a model, it’s common to examine the <strong>loss curve</strong> to check whether the model has converged or to observe its learning behavior. The loss curve illustrates how the loss changes over the course of training. Below is an example of what a typical loss curve looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*FjRcX5tFn8OC2s6QpoS-oQ.png" /><figcaption>Loss Curve</figcaption></figure><p>Notice how the loss drops sharply during the initial iterations, then continues to decrease more gradually, eventually leveling off around the 1,000th iteration. Once it flattens out at this point, we can be fairly confident that the model has converged.</p><p>Following is how the model has changed over the training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*FMdi3Xop6QgD8SGZoG2RTg.png" /><figcaption>Model Training</figcaption></figure><h4>Conclusion</h4><p>Loss gives us a way to measure just how wrong a model’s predictions are, while gradient descent provides the mechanism to systematically reduce that error, one small step at a time. By tracking the loss curve during training, we can tell whether a model is genuinely learning or has already converged. Together, these two concepts turn a model with random, meaningless parameters into one that can make increasingly accurate predictions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43e90ebb9c36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/understanding-loss-and-gradient-descent-how-a-model-learns-43e90ebb9c36">Understanding Loss and Gradient Descent: How a Model Learns</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression Explained]]></title>
            <link>https://medium.com/h7w/linear-regression-explained-b6d94244876a?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6d94244876a</guid>
            <category><![CDATA[numerical-data]]></category>
            <category><![CDATA[ai-model]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 02:11:44 GMT</pubDate>
            <atom:updated>2026-08-24T02:11:44.577Z</atom:updated>
            <content:encoded><![CDATA[<p>We’ll build the intuition behind linear regression, understand how a model learns its parameters, and see what the training process looks like.</p><p>Linear regression is used to find relationship between <em>features </em>and a <em>label. </em>Linear regression model works with numbers so features and label should be numerical values.</p><p>Features are inputs or independent variables that are used to predict label (output value)</p><p>Suppose we have historical data about the house size and its price in thousand dollars. Given the size of a new house, we want to predict how much it will cost.</p><p>This is a regression problem because the output is a continuous numerical value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*vazQUXsqpToFm1rokCrghw.png" /><figcaption>House Dataset</figcaption></figure><p>Plotting these point would give the following graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*CkAZq9FMZPFiR0fo.png" /></figure><p>A model can be created by drawing a best fit line through the points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*8c6Bdg6mD8qeJDjg.png" /></figure><h4>Linear regression equation</h4><p>The model would be mathematiclly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/96/0*-XoXqTjs7-OeL7UQ.png" /></figure><ul><li><em>y</em> house price — the value we want to predict.</li><li><em>m </em>is the slope of the line.</li><li><em>x</em> is house size — input value.</li><li><em>b</em> is the y-intercept.</li></ul><p>In ML terms, the linear regression model would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/107/0*Pj6Hic72BwQxR0n5.png" /></figure><ul><li><em>y</em> is the predicted label — the output.</li><li><em>b</em> is the bias of the model. Bias is a parameter of the model and is calculated during training.</li><li><em>w</em>1​ is the weight of the feature. Weight is a parameter of the model and is calculated during training.</li><li><em>x</em>1​ is a feature — the input.</li></ul><p>weight and bias are learned during training to produce the best model.</p><p>From the model or line we made, the bias is 65.79 (where the line intersects at y) and the weight is 0.1565 (slope of the line)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*QsO3Whbun6VBczmz.png" /></figure><p>The model can be defined from this equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/191/0*0jKXalpa-EHu0tJM.png" /></figure><p>This equation can be used to make predictions, we would need the input value (house size) to find the output (house price) according to the model we built.</p><h4>Models with multiple features</h4><p>Although the example in this section uses only one feature. In real world, the house price depend on multiple features like no. of floors, area, location, no. of basic amenities, etc. For example, a model that takes five features would be written as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*uis5wkxM5YfxSn7R.png" /></figure><h4>Conclusion</h4><p>By learning the weights and bias from historical examples, the model can represent the relationship between features and a label as a line or linear equation. While our example used only house size, the same idea can be extended to multiple features, allowing the model to capture more complex real-world relationships.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6d94244876a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/h7w/linear-regression-explained-b6d94244876a">Linear Regression Explained</a> was originally published in <a href="https://medium.com/h7w">T3CH</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Target Encoding Strategies for Categorical Features]]></title>
            <link>https://medium.com/@alihamzabham/target-encoding-strategies-for-categorical-features-553b6e29e8c2?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/553b6e29e8c2</guid>
            <category><![CDATA[smoothing]]></category>
            <category><![CDATA[high-cardinality]]></category>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[target-encoding]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 07:24:38 GMT</pubDate>
            <atom:updated>2026-08-19T07:24:38.919Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgtOxH5YcqADGRW0HPQBLg.jpeg" /><figcaption>Target Encoding Strategies for Categorical Features</figcaption></figure><p>A dive into target encoding and smoothing, how to turn high-cardinality categories into powerful numeric features.</p><p>Previously, we’ve covered <a href="https://medium.com/@alihamzabham/feature-engineering-foundations-feature-selection-and-simple-transforms-ef53a4be877f?source=user_profile_page---------1-------------d86d32b6eb9a----------------------">Baseline scoring, mutual information, and simple transforms</a> and <a href="https://medium.com/@alihamzabham/advanced-data-reshaping-with-k-means-and-pca-d19cf44fc34a?source=user_profile_page---------0-------------d86d32b6eb9a----------------------">K-Means clustering and PCA</a> respectively. In this article, we turn to a common but tricky problem: encoding <strong>categorical features</strong>, especially ones with many unique categories.</p><h4>The Problem with High-Cardinality Categoricals</h4><p>Most feature engineering techniques we’ve covered so far work on numerical features. But real-world datasets contain categorical columns too.</p><p>Consider a <em>StoreID </em>column in our retail dataset with hundreds or thousand of unique store locations. Standard encoding approaches struggle here:</p><ul><li><strong>One-hot encoding </strong>would create thousands of new columns, most of them mostly zeros. Resulting in bloating your dataset and hurting model performance.</li><li><strong>Label encoding </strong>(assigning each category an arbitrary integer) implies a false ordinal relationship between categories that don’t actually have one.</li></ul><p>This is where <strong>Target Encoding </strong>comes in.</p><h4>What Is Target Encoding?</h4><p><strong>Target encoding</strong> replaces each category with a number <em>derived from the target variable itself</em>. Because it uses the target to build the encoding, it’s a <strong>supervised</strong> feature engineering technique.</p><p>The simplest and most common version is a <strong>mean encoding</strong>: replace each category with the average target value for that category.</p><pre>import pandas as pd<br><br>df = pd.read_csv(&quot;retail_orders.csv&quot;)<br><br># Replace each StoreID with the average OrderProfit for that store<br>df[&quot;StoreID_encoded&quot;] = df.groupby(&quot;StoreID&quot;)[&quot;OrderProfit&quot;].transform(&quot;mean&quot;)<br><br>df[[&quot;StoreID&quot;, &quot;OrderProfit&quot;, &quot;StoreID_encoded&quot;]].head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*fDjOIwpDNytaTYDTiTyczQ.png" /><figcaption>Target Encoding</figcaption></figure><p>This kind of encoding is sometimes called a <strong>mean encoding</strong>. Applied to a binary target, it’s also known as <strong>bin counting</strong>. You may also see it referred to as likelihood encoding, impact encoding, or leave-one-out encoding, depending on the exact implementation.</p><h4>The Problem with Naive Target Encoding</h4><p>A raw mean encoding like the one above has two significant weaknesses:</p><ol><li><strong>Unknown categories </strong>If a new <em>StoreID </em>shows up in production data that wasn’t in the training set, there’s no encoded value for it.</li><li><strong>Rare categories </strong>A store with only two or three orders in the training data gets an encoded value based on a tiny, unreliable sample. The model can end up overfitting to noise in these rare categories rather than learning a real pattern.</li></ol><h4>The Solution: Smoothing</h4><p>The idea is to blend the <em>in-category</em> average with the <em>overall</em> average. Rare categories get less weight on their category average, while missing categories just get the overall average.</p><pre>weight = n / (n + m)<br>encoding = weight * in_category_average + (1 - weight) * overall_average</pre><p>Where:</p><ul><li><em>n </em>is the number of times that category appears in the data</li><li><em>m </em>is a tunable <strong>smoothing factor</strong>,<strong> </strong>larger values of <em>m </em>put more weight on the overall average</li></ul><p>Categories with lots of data get an encoding close to their true in-category average. Categories with little data or that are entirely missing fall back toward the overall average instead of an unreliable (or nonexistent) category-specific value.</p><h4>When to Use Target Encoding</h4><p>Target encoding is most valuable when:</p><ul><li>A categorical feature has <strong>high cardinality </strong>(many unique values), making one-hot encoding impractical.</li><li>The category is likely to have a genuine relationship with the target (e.g., some stores are simply more profitable than others).</li><li>You have enough data per category to compute a reasonably reliable average, and you’re using smoothing to handle the categories that don’t.</li></ul><p>It’s less useful for low-cardinality features (a handful of categories, like <em>CustomerSegment</em>), where one-hot encoding works fine and doesn’t carry the same overfitting risk.</p><p><strong>A Worked Example</strong></p><p>Suppose our dataset has only 4 orders from <em>StoreID = “STORE_112”,</em> with an average profit of $18. The overall average profit across all stores is $45. If we choose <em>m = 6</em></p><pre>n = 4<br>m = 6<br>weight = n / (n + m) # = 4 / 10 = 0.4<br><br>encoding = weight * 18 + (1 - weight) * 45<br># = 0.4 * 18 + 0.6 * 45 = 34.2</pre><p>So instead of trusting the noisy $18 average from just four orders, the smoothed encoding pulls the value up toward the more reliable overall average, landing at $34.20. A store with hundreds of orders, by contrast, would end up with an encoding very close to its own true average, since <em>weight </em>approaches 1 as <em>n </em>grows.</p><p>Implementing Smoothed Target Encoding</p><p>The <em>category_encoders </em>library provides a ready-made implementation via <em>MEstimateEncoder</em>:</p><pre>from category_encoders import MEstimateEncoder<br><br>X = df.copy()<br>y = X.pop(&quot;OrderProfit&quot;)<br><br># Split off a portion of the data purely for fitting the encoder<br>X_encode = X.sample(frac=0.25, random_state=0)<br>y_encode = y[X_encode.index]<br><br>X_train = X.drop(X_encode.index)<br>y_train = y[X_train.index]<br><br># Create the encoder. Choose m to control how aggressively rare<br># categories get pulled toward the overall average.<br>encoder = MEstimateEncoder(cols=[&quot;StoreID&quot;], m=5.0)<br><br># Fit the encoder on the held-out encoding split<br>encoder.fit(X_encode, y_encode)<br><br># Apply the encoding to the training data<br>X_train = encoder.transform(X_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*OKvGwheF00Uytc-L2jCMmw.png" /><figcaption>Encoding Smoothing</figcaption></figure><p><strong>Why Split the Data First?</strong></p><p>Notice that we fit the encoder on a separate <em>X_encode </em>split rather than the full training set. This matters: if you compute a target encoding using the <em>same</em> rows you’ll train your model on, you leak information about the target directly into a feature the model sees during training. That inflates validation scores in a way that won’t hold up on genuinely new data.</p><p>Splitting off a portion of the data purely for fitting the encoder — and applying the resulting encoding to the rest keep the encoding honest and generalizable.</p><p><strong>Visualizing the Encoding</strong></p><p>It’s worth checking that your encoded feature’s distribution resembles the target’s distribution, that’s a sign the encoding is capturing real signal rather than noise.</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>plt.figure(dpi=90)<br>ax = sns.histplot(y, stat=&quot;density&quot;, ax=None)<br>ax = sns.kdeplot(X_train[&quot;StoreID&quot;], color=&quot;r&quot;, ax=ax)<br>ax.set_xlabel(&quot;OrderProfit&quot;)<br>ax.legend(labels=[&quot;StoreID (encoded)&quot;, &quot;OrderProfit&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*vFXaHJmcl7OkTo7MYoUeZA.png" /><figcaption>Encoding Visualization</figcaption></figure><p>If the red curve (encoded <em>StoreID</em>) tracks the shape of the actual target distribution reasonably well, that’s a good sign the encoding is meaningful rather than overfit.</p><h4>Conclusion</h4><ul><li><strong>Target encoding </strong>replaces categories with a number derived from the target, making it a supervised technique well suited to high-cardinality categorical features.</li><li>Naive target encoding overfits on rare categories and breaks on unseen ones, <strong>smoothing</strong></li><li>fixes both by blending category-level and overall averages.</li><li>Always <strong>fit the encoder on a separate data split </strong>from the one you train your final model on, to avoid target leakage.</li><li>Validate your encoded feature by comparing its distribution to the target’s, and by checking whether it improves your cross-validated model score.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=553b6e29e8c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced Data Reshaping with K-Means and PCA]]></title>
            <link>https://medium.com/@alihamzabham/advanced-data-reshaping-with-k-means-and-pca-d19cf44fc34a?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d19cf44fc34a</guid>
            <category><![CDATA[data-reshaping]]></category>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[model-training]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 13:27:17 GMT</pubDate>
            <atom:updated>2026-08-13T13:27:17.260Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@alihamzabham/feature-engineering-foundations-feature-selection-and-simple-transforms-ef53a4be877f">previous article</a>, we covered baseline scoring, mutual information, and simple transforms like ratios, counts, and group aggregations. Now, we’ll use two unsupervised learning algorithms K-means clustering and PCA for feature discovery.</p><h3>Why Unsupervised Learning for Feature Engineering?</h3><p>Simple transforms (ratios, logs, counts) work well when you already have an idea of what relationship might matter. But sometimes the useful structure in your data isn’t obvious from individual columns, it only emerges when you look at how <em>multiple</em> features vary together.</p><p>This is where unsupervised algorithms come in. You can think of an algorithm like K-Means or PCA as a <strong>“feature discovery” </strong>technique: instead of predicting a target, it finds structure in the data itself, which you can then feed back into your model as a new feature.</p><h3>Clustering with K-Means</h3><p><strong>K-Means clustering</strong> groups similar rows together based on their feature values, assigning each row a cluster label. That label can itself be a powerful new feature, especially when the underlying groups behave differently with respect to your target.</p><p><strong>Preparing the Data</strong></p><p>K-Means is distance-based, so features need to be on the same scale before clustering. Otherwise, features with larger numeric ranges will dominate the distance calculation.</p><pre>import pandas as pd<br>from sklearn.cluster import KMeans<br><br>df = pd.read_csv(&quot;retail_orders.csv&quot;)<br><br>X = df.copy()<br>y = X.pop(&quot;OrderProfit&quot;)<br><br># Choose the features to use for clustering<br>features = [&quot;UnitPrice&quot;, &quot;Quantity&quot;, &quot;ShippingCost&quot;, &quot;Discount&quot;]<br><br># Standardize the features (mean = 0, std = 1)<br>X_scaled = X.loc[:, features]<br>X_scaled = (X_scaled - X_scaled.mean(axis=0)) / X_scaled.std(axis=0)</pre><p><strong>Creating a Cluster Label Feature</strong></p><pre>kmeans = KMeans(n_clusters=6, n_init=10, random_state=0)<br>X[&quot;Cluster&quot;] = kmeans.fit_predict(X_scaled)<br>X[&quot;Cluster&quot;] = X[&quot;Cluster&quot;].astype(&quot;category&quot;)<br><br>X.head()</pre><p>This single <em>Cluster </em>column might now separate, for example, “high-volume, low-discount orders” from “low-volume, heavily discounted orders”, a distinction that could take several manually engineered features to reconstruct otherwise.</p><p><strong>Cluster-Distance Features</strong></p><p>Rather than assigning each row to a single cluster, you can also measure <strong>how far each row is from every cluster’s center</strong>. This produces one new numeric feature per cluster, capturing more nuance than a single categorical label.</p><pre>kmeans = KMeans(n_clusters=6, n_init=10, random_state=0)<br><br># fit_transform (instead of fit_predict) returns distances to each centroid<br>X_cd = kmeans.fit_transform(X_scaled)<br><br># Label the new features and join them to the dataset<br>X_cd = pd.DataFrame(<br>    X_cd, columns=[f&quot;Centroid_{i}&quot; for i in range(X_cd.shape[1])]<br>)<br>X = X.join(X_cd)</pre><p>Now, instead of a single <em>Cluster </em>label, you have several <em>Centroid_0, Centroid_1, … </em>features, each describing how close an order is to a particular “type” of order in your dataset. This can be especially useful for models that benefit from continuous inputs rather than categorical ones.</p><p><strong>Choosing the Number of Clusters</strong></p><p>There’s no universally correct number of clusters, it depends on your data and problem. A common starting approach is to try a range of values (e.g., 3 to 10) and evaluate how each affects your model’s cross-validation score, rather than relying purely on visual inspection.</p><h3>Principal Component Analysis (PCA)</h3><p>Where K-Means groups similar rows together, <strong>PCA</strong> takes a different approach: it re-describes your <em>features</em> along new axes that capture the directions of greatest variation in the data.</p><p><strong>The Core Idea</strong></p><p>Imagine two correlated features, <em>UnitPrice </em>and <em>ShippingCost, </em>that tend to increase together. Instead of describing each order using both original features, PCA might describe it using:</p><ul><li><strong>PC1 (“Order Scale”)</strong>: a combination that captures overall order size/value.</li><li><strong>PC2 (“Price-to-Shipping Balance”)</strong>: a combination that captures how the two features diverge from each other.</li></ul><p>These new features called <strong>principal components</strong> are just weighted sums (linear combinations) of the original features:</p><pre>df[&quot;OrderScale&quot;] = 0.707 * X[&quot;UnitPrice&quot;] + 0.707 * X[&quot;ShippingCost&quot;]<br>df[&quot;PriceShippingBalance&quot;] = 0.707 * X[&quot;UnitPrice&quot;] - 0.707 * X[&quot;ShippingCost&quot;]</pre><p>The weights themselves are called <strong>loadings</strong>, and there will always be as many principal components as there are original features.</p><p><strong>Computing PCA in Practice</strong></p><pre>from sklearn.decomposition import PCA<br><br># Reuse the same standardized features from the clustering step<br>pca = PCA()<br>X_pca = pca.fit_transform(X_scaled)<br><br>component_names = [f&quot;PC{i+1}&quot; for i in range(X_pca.shape[1])]<br>X_pca = pd.DataFrame(X_pca, columns=component_names)<br><br>X_pca.head()</pre><p>To understand what each component actually represents, inspect the loadings:</p><pre>loadings = pd.DataFrame(<br>    pca.components_.T,          # transpose so components become columns<br>    columns=component_names,<br>    index=X_scaled.columns,     # original feature names<br>)<br>loadings</pre><p>If, for example, <em>UnitPrice </em>and <em>ShippingCost </em>both load heavily and with the same sign on PC1, that component represents “overall order scale.” If <em>Discount </em>loads with the opposite sign of <em>Quantity </em>on PC2, that component captures a “bulk-vs-discount” trade-off pattern.</p><p><strong>When PCA Helps</strong></p><p>PCA is useful for feature engineering in several distinct ways:</p><ul><li><strong>Dimensionality reduction</strong>,<strong> </strong>PCA concentrates redundancy into one or more near-zero-variance components, which can often be dropped with little information loss.</li><li><strong>Anomaly detection</strong>,<strong> </strong>Unusual combinations of features can show up as outliers in a principal component that wouldn’t be visible in any single original feature.</li><li><strong>Noise reduction</strong>,<strong> </strong>PCA can concentrate the informative signal into a smaller number of components while leaving noise behind in the rest.</li><li><strong>Decorrelation</strong>,<strong> </strong>PCA transforms correlated features into uncorrelated components, which can be easier for some algorithms (particularly linear models) to work with.</li></ul><p><strong>Checking Component Usefulness with Mutual Information</strong></p><p>you can score the new PCA-derived features with <a href="https://medium.com/@alihamzabham/feature-engineering-foundations-feature-selection-and-simple-transforms-ef53a4be877f">mutual information</a> to see which components are actually informative:</p><pre>from sklearn.feature_selection import mutual_info_regression<br>def make_mi_scores(X, y, discrete_features):<br>    mi_scores = mutual_info_regression(X, y, discrete_features=discrete_features)<br>    mi_scores = pd.Series(mi_scores, name=&quot;MI Scores&quot;, index=X.columns)<br>    return mi_scores.sort_values(ascending=False)<br>mi_scores = make_mi_scores(X_pca, y, discrete_features=False)<br>print(mi_scores)</pre><h3>Conclusion</h3><ul><li><strong>K-Means clustering </strong>turns groups of similar rows into a new categorical feature (or a set of distance-based numeric features), surfacing structure that isn’t obvious from individual columns.</li><li><strong>PCA </strong>re-expresses your features along their axes of greatest variation, which is useful for dimensionality reduction, decorrelation, noise reduction, and anomaly detection.</li><li>Both techniques require <strong>standardized (scaled) input features </strong>to work correctly.</li><li>Always validate new cluster or component features with mutual information or a cross-validated model score.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d19cf44fc34a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Engineering: Foundations, Feature Selection, and Simple Transforms]]></title>
            <link>https://medium.com/@alihamzabham/feature-engineering-foundations-feature-selection-and-simple-transforms-ef53a4be877f?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef53a4be877f</guid>
            <category><![CDATA[prediction-model]]></category>
            <category><![CDATA[creating-features]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[mutual-information]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 20:16:46 GMT</pubDate>
            <atom:updated>2026-08-11T20:16:46.822Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ayWUoU51LBLT5U7jqp3LdA.jpeg" /><figcaption>Feature Engineering: Foundations, Feature Selection, and Simple Transforms</figcaption></figure><h3>What is Feature Engineering?</h3><p>A machine learning model is only as good as the data you feed it. <strong>Feature engineering</strong> is the process of using domain knowledge to create new input variables (features) that make patterns easier for a model to learn. Done well, it often improves model performance more than switching algorithms or tuning hyperparameters ever could.</p><p>This article covers:</p><ul><li>Establishing a <strong>baseline score </strong>before adding any new features</li><li>Using <strong>mutual information </strong>to rank which features are worth developing</li><li>Building simple, high-value features through <strong>math transforms, counts, and string operations</strong></li></ul><p>Throughout, we’ll use a single running example: an <strong>online retail orders dataset</strong>, with the goal of predicting <em>OrderProfit </em>(the profit generated by each order). Using one consistent dataset across the whole series makes it easier to see how each technique builds on the last.</p><p>Our starting columns look like this:</p><ul><li><strong>UnitPrice </strong>—<strong> </strong>Price per unit sold</li><li><strong>Quantity </strong>—<strong> </strong>Number of units in the order</li><li><strong>ShippingCost</strong> — Cost to ship the order</li><li><strong>Discount</strong> — Discount applied, represented as a fraction</li><li><strong>CustomerSegment</strong> — Categorical variable: Consumer, Corporate, or Home Office</li><li><strong>OrderDate</strong> — Timestamp of the order</li><li><strong>Region</strong> — Categorical variable: North, South, East, or West</li><li><strong>OrderProfit</strong> — Target variable</li></ul><h4>1. Establish a Baseline Score</h4><p>We’ll first establish a baseline by training the model on the raw dataset. This will help us determine whether our new features are actually useful.</p><p>A baseline score can help you decide whether your new features are worth keeping, or whether you should discard them and possibly try something else.</p><pre>import pandas as pd<br>from sklearn.ensemble import RandomForestRegressor<br>from sklearn.model_selection import cross_val_score<br><br>df = pd.read_csv(&quot;retail_orders.csv&quot;)<br><br>X = df.copy()<br>y = X.pop(&quot;OrderProfit&quot;)<br><br># Train and score baseline model<br>baseline = RandomForestRegressor(criterion=&quot;absolute_error&quot;, random_state=0)<br>baseline_score = cross_val_score(<br>    baseline, X, y, cv=5, scoring=&quot;neg_mean_absolute_error&quot;<br>)<br>baseline_score = -1 * baseline_score.mean()<br><br>print(f&quot;MAE Baseline Score: {baseline_score:.4}&quot;)</pre><p>This gives you a concrete number (mean absolute error, in this case) to compare against every time you engineer a new feature. No baseline, no way to know if your work is actually helping.</p><h4>2. Rank Features with Mutual Information</h4><p>Once you have a baseline, the next question is: <strong>which features actually matter or have the most potential? </strong>Rather than guessing, you can use a <em>feature utility metric.</em></p><p><strong>Feature Utility Metric</strong>: a function that scores how strongly each feature is associated with the target.</p><p><strong>Mutual information (MI)</strong> is one of the best general-purpose metrics for this:</p><ul><li>It detects any<strong> </strong>kind of relationship (not just linear, like correlation does)</li><li>It’s computationally efficient and easy to interpret</li><li>It’s resistant to overfitting</li><li>An MI score of zero means the feature and target are independent — knowing one tells you nothing about the other</li></ul><p>There’s one important limitation to keep in mind: MI is a <strong>univariate</strong> metric. It evaluates each feature <em>on its own</em>, so it can’t detect interaction effects between features. A feature with a low MI score might still be valuable in combination with others, don’t discard it purely on that basis without checking for interactions with domain knowledge or visualization.</p><pre>from sklearn.feature_selection import mutual_info_regression<br><br>X = df.copy()<br>y = X.pop(&quot;OrderProfit&quot;)<br><br># Label encode categorical features<br>for colname in X.select_dtypes(&quot;object&quot;):<br>    X[colname], _ = X[colname].factorize()<br><br># Mark which features are discrete (MI treats these differently)<br>discrete_features = X.dtypes == int<br><br>def make_mi_scores(X, y, discrete_features):<br>    mi_scores = mutual_info_regression(X, y, discrete_features=discrete_features)<br>    mi_scores = pd.Series(mi_scores, name=&quot;MI Scores&quot;, index=X.columns)<br>    mi_scores = mi_scores.sort_values(ascending=False)<br>    return mi_scores<br><br>mi_scores = make_mi_scores(X, y, discrete_features)<br>print(mi_scores)</pre><p>Visualizing the scores makes it easy to spot your best candidates for further development:</p><pre>import matplotlib.pyplot as plt<br>import numpy as np<br><br>def plot_mi_scores(scores):<br>    scores = scores.sort_values(ascending=True)<br>    width = np.arange(len(scores))<br>    ticks = list(scores.index)<br>    plt.barh(width, scores)<br>    plt.yticks(width, ticks)<br>    plt.title(&quot;Mutual Information Scores&quot;)<br><br>plt.figure(dpi=100, figsize=(8, 5))<br>plot_mi_scores(mi_scores)</pre><p>In our example, <em>Quantity </em>and <em>UnitPrice </em>typically show the highest MI scores against <em>OrderProfit</em>, while <em>Region </em>might rank lower, but that doesn’t automatically make <em>Region </em>useless. <em>MI tells you where to start, not the full story.</em></p><h4>3. Create Simple, High-Value Features</h4><p>With a ranked list of promising features, you can start engineering new ones. The most effective new features are often the simplest.</p><p><strong>Math Transforms: Ratios and Combinations<br></strong>Just like a recipe’s outcome often depends more on the <em>ratio</em> of ingredients than their absolute amounts, ratios between features are frequently more predictive than the raw values alone.</p><pre>X = df.copy()<br>y = X.pop(&quot;OrderProfit&quot;)<br><br># Create ratio-based features<br>X[&quot;DiscountedPrice&quot;] = X[&quot;UnitPrice&quot;] * (1 - X[&quot;Discount&quot;])<br>X[&quot;ShippingToPriceRatio&quot;] = X[&quot;ShippingCost&quot;] / X[&quot;UnitPrice&quot;]<br>X[&quot;RevenuePerUnit&quot;] = (X[&quot;UnitPrice&quot;] * X[&quot;Quantity&quot;]) / X[&quot;Quantity&quot;]<br><br>model = RandomForestRegressor(criterion=&quot;absolute_error&quot;, random_state=0)<br>score = cross_val_score(model, X, y, cv=5, scoring=&quot;neg_mean_absolute_error&quot;)<br>score = -1 * score.mean()<br><br>print(f&quot;MAE Score with Ratio Features: {score:.4}&quot;)</pre><p>Keep new features as simple as possible. The more complicated a formula, the harder it is for a model and for you to make sense of it.</p><p><strong>Reshaping Skewed Features</strong><br>Data visualization often reveals that a feature is heavily skewed, which can make it harder for a model to learn from. Applying a <strong>power or logarithmic transform</strong> can normalize the distribution.</p><pre>import seaborn as sns<br><br># ShippingCost is right-skewed — a log transform helps normalize it<br># Use log1p (log(1+x)) instead of log to safely handle zero values<br>df[&quot;LogShippingCost&quot;] = df[&quot;ShippingCost&quot;].apply(np.log1p)<br><br>fig, axs = plt.subplots(1, 2, figsize=(8, 4))<br>sns.kdeplot(df[&quot;ShippingCost&quot;], fill=True, ax=axs[0])<br>sns.kdeplot(df[&quot;LogShippingCost&quot;], fill=True, ax=axs[1])</pre><p><strong>Counting Related Features</strong><br>When several columns describe related yes/no conditions, aggregating them into a single <strong>count</strong> feature can be more informative than the individual flags.</p><pre>promo_flags = [&quot;FreeShippingFlag&quot;, &quot;BundleDealFlag&quot;, &quot;LoyaltyDiscountFlag&quot;, &quot;SeasonalSaleFlag&quot;]<br>df[&quot;PromotionsApplied&quot;] = df[promo_flags].sum(axis=1)</pre><p>This single PromotionsApplied column tells the model &quot;how many promotions touched this order&quot; in one number, rather than forcing it to learn the same pattern across four separate binary columns.</p><p><strong>Building Up and Breaking Down Structured Text</strong><br>Fields like order IDs, addresses, or timestamps often have internal structure worth extracting. Pandas’ .str accessor makes this easy.</p><pre># OrderDate looks like: &quot;2024-03-15 14:22:00&quot;<br>df[&quot;OrderYear&quot;] = df[&quot;OrderDate&quot;].str.split(&quot;-&quot;).str[0].astype(int)<br>df[&quot;OrderMonth&quot;] = df[&quot;OrderDate&quot;].str.split(&quot;-&quot;).str[1].astype(int)</pre><p>Breaking a compound field into its components often surfaces patterns (like seasonality) that were invisible in the original format.</p><p><strong>Group Transforms</strong><br>Finally, <strong>group transforms</strong> let you aggregate information across rows that share a category. For example, “the average profit for a customer segment” or “the typical shipping cost for a region.” To compute this in Pandas, we use the <em>groupby </em>and <em>transform </em>methods:</p><pre>df[&quot;AvgProfitBySegment&quot;] = (<br>    df.groupby(&quot;CustomerSegment&quot;)[&quot;OrderProfit&quot;]<br>    .transform(&quot;mean&quot;)<br>)</pre><p>Other handy methods include <em>max</em>, <em>min</em>, <em>median</em>, <em>std</em>, and <em>count.</em></p><h4>Conclusion</h4><ul><li>Always start with a <strong>baseline score</strong>,<strong> </strong>it’s your only reliable way to know if a new feature is actually helping.</li><li><strong>Mutual information</strong>,<strong> </strong>is a fast, model-agnostic way to rank features by potential, but remember it can’t detect interaction effects.</li><li>The most effective new features are often the simplest: <strong>ratios, log transforms, counts, string splits, and group aggregations</strong></li><li>Re-score your model after every meaningful batch of new features to confirm they’re adding value.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef53a4be877f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Fix Character Encoding Problems in Your Data]]></title>
            <link>https://medium.com/@alihamzabham/how-to-fix-character-encoding-problems-in-your-data-6f266f999788?source=rss-d86d32b6eb9a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f266f999788</guid>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[utf-8]]></category>
            <category><![CDATA[character-encoding]]></category>
            <category><![CDATA[csv]]></category>
            <dc:creator><![CDATA[Hamza Ahmed]]></dc:creator>
            <pubDate>Sat, 08 Aug 2026 06:06:36 GMT</pubDate>
            <atom:updated>2026-08-08T06:06:36.103Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GtiBjNKpij14CP4TaT06sQ.jpeg" /></figure><p>If you’ve ever opened a CSV file and seen garbled text like “Ã©” where an é should be, or hit a <em>UnicodeDecodeError</em> while trying to read in a dataset, you’ve run into a character encoding problem. This is one of the most common and most confusing issues in data cleaning. This guide walks through what encodings actually are, how to diagnose problems, and how to fix them.</p><h3>What is a character encoding?</h3><p>Under the hood, every file on your computer is just a string of bytes — raw binary data that looks something like <em>0110100001101001</em>. A <strong>character encoding</strong> is a set of rules for mapping those raw bytes to the human-readable characters that make up text (like “hi”).</p><p>There are many different character encodings in use, each with its own rules for this mapping. The one you’ll run into most often, and the one you should default to whenever possible, is <strong>UTF-8</strong>. It can represent virtually any character from any language, and it’s the standard encoding assumed by most modern tools.</p><h3>Strings vs. bytes in Python</h3><p>When working with text in Python 3, you’ll deal with two related but distinct data types:</p><ul><li><strong>str </strong>— the default text type. This is what you get when you type text directly, like “hi”.</li><li><strong>bytes </strong>— the raw, encoded representation of that text.</li></ul><p>You move between them with two methods: <em>.encode()</em> turns a string into bytes (using a specified encoding), and <em>.decode()</em> turns bytes back into a string.</p><pre>before = &quot;This is the euro symbol: €&quot;  # a normal Python string (UTF-8 under the hood)<br><br># encode it to a different encoding, replacing characters that raise errors<br>after = before.encode(&quot;utf-8&quot;, errors=&quot;replace&quot;)<br><br># check the type<br>type(after)  # &lt;class &#39;bytes&#39;&gt;<br><br># convert it back to a string<br>print(after.decode(&quot;utf-8&quot;))<br># This is the euro symbol: €</pre><p>Here, encoding to UTF-8 and decoding from UTF-8 round-trips cleanly, because the € symbol is fully supported by UTF-8.</p><h3>What happens when the encoding doesn’t match</h3><p>Problems appear when you try to encode text into a character set that doesn’t support all of its characters. For example, ASCII, an older, more limited encoding has no way to represent the euro symbol at all.</p><pre>before = &quot;This is the euro symbol: €&quot;  # originally a UTF-8 string in Python<br><br># encode it to ASCII, replacing characters that raise errors<br>after = before.encode(&quot;ascii&quot;, errors=&quot;replace&quot;)<br><br># convert it back to a string<br>print(after.decode(&quot;ascii&quot;))<br># This is the euro symbol: ?</pre><p>Notice what happened: the € was silently replaced with ?. Once that substitution happens, <strong>the original byte string is gone for good</strong>. There’s no way to recover the euro symbol from the ?. This is why encoding mistakes are so dangerous during data cleaning: they don’t always throw an error, they sometimes just quietly destroy information.</p><p>This is also why the errors=”replace” argument is worth using with caution. It’s useful for spotting <em>where</em> problems exist, but it’s a lossy operation, don’t use it as your final cleaning step unless you’re okay losing that data.</p><p>The best time to convert non-UTF-8 input into UTF-8 is right when you read a file in, before any other processing happens.</p><h3>Reading in files with encoding problems</h3><p>Most files you’ll encounter are encoded in UTF-8, which is also what Python expects by default. So, most of the time, reading in data just works. But every so often, you’ll hit an error like this:</p><pre># try to read in a file that isn&#39;t UTF-8<br>kickstarter_2016 = pd.read_csv(&quot;../input/kickstarter-projects/ks-projects-201612.csv&quot;)<br># UnicodeDecodeError: &#39;utf-8&#39; codec can&#39;t decode byte 0x96 in position ...</pre><p>This error means the file was saved with some other encoding, and Python’s default UTF-8 assumption doesn’t fit the bytes it’s reading.</p><h3>Figuring out the real encoding</h3><p>One option is brute force: try a handful of common encodings (“latin-1”, “Windows-1252”, “iso-8859–1”, and so on) until one works. A faster, more reliable option is to let a library guess for you. The charset_normalizer package (a modern, actively maintained alternative to the older chardet) inspects a sample of the raw bytes and estimates which encoding produced them.</p><pre>import charset_normalizer<br><br># look at the first ten thousand bytes to guess the character encoding<br>with open(&quot;../input/kickstarter-projects/ks-projects-201801.csv&quot;, &#39;rb&#39;) as rawdata:<br>    result = charset_normalizer.detect(rawdata.read(10000))<br><br># check what the character encoding might be<br>print(result)<br># {&#39;encoding&#39;: &#39;Windows-1252&#39;, &#39;confidence&#39;: 0.73, &#39;language&#39;: &#39;&#39;}</pre><p>A few notes on this:</p><ul><li>We only read the first 10,000 bytes rather than the whole file. Detection works on a sample, and for large files, reading the whole thing just to guess the encoding wastes time and memory.</li><li>The result includes a <strong>confidence score</strong>. Detection is a statistical guess, not a certainty. Always sanity-check the output against the actual data, especially when confidence is low.</li><li>The file has to be opened in binary mode (‘rb’) for this to work, since the detector needs the raw, undecoded bytes.</li></ul><p>Once you have a likely encoding, pass it to pandas explicitly:</p><pre># read in the file with the encoding detected by charset_normalizer<br>kickstarter_2016 = pd.read_csv(<br>    &quot;../input/kickstarter-projects/ks-projects-201612.csv&quot;,<br>    encoding=&#39;Windows-1252&#39;<br>)<br><br># look at the first few lines<br>kickstarter_2016.head()</pre><p>If the file still doesn’t read in cleanly, or the text looks subtly wrong (accented characters turned into odd symbols, for instance), it’s worth trying a couple of the encoding’s close relatives, Windows-1252 and iso-8859–1 in particular are similar enough to be mistaken for one another but differ on a handful of byte values.</p><h3>Saving your files with UTF-8 encoding</h3><p>Once you’ve successfully loaded and cleaned a file, there’s no reason to carry an unusual encoding forward. UTF-8 is the standard format in Python, and pandas uses it by default when writing files:</p><pre># save our file (will be saved as UTF-8 by default!)<br>kickstarter_2016.to_csv(&quot;ks-projects-201801-utf8.csv&quot;)</pre><p>This means that even if you had to jump through some hoops to <em>read</em> a messy file, the version you save and pass along to teammates or downstream pipelines can be clean, standard UTF-8, sparing the next person the same debugging session.</p><h3>Closing Thoughts</h3><ul><li><strong>Default to UTF-8 </strong>whenever you’re creating or saving files. It’s the modern standard and handles the widest range of characters.</li><li><strong>Convert encodings at read time</strong>, not partway through your pipeline — the earlier you standardize, the fewer surprises later.</li><li><strong>Be careful with errors=”replace”</strong>,<strong> </strong>it hides problems rather than fixing them, and the replaced data can’t be recovered.</li><li><strong>Use charset_normalizer.detect() </strong>on a byte sample to guess an unknown file’s encoding, but treat the result as a starting point, not a guarantee.</li><li><strong>Always save your cleaned output as UTF-8 </strong>so encoding issues don’t propagate downstream.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f266f999788" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>