<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Cainã Max Couto da Silva on Medium]]></title>
        <description><![CDATA[Stories by Cainã Max Couto da Silva on Medium]]></description>
        <link>https://medium.com/@cmcouto.silva?source=rss-5da4fd61571c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k8oeLc9zGNIMT51kQFaOtA.png</url>
            <title>Stories by Cainã Max Couto da Silva on Medium</title>
            <link>https://medium.com/@cmcouto.silva?source=rss-5da4fd61571c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:36:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cmcouto.silva/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The categorical encoder you choose might impact the model performance]]></title>
            <link>https://medium.com/@cmcouto.silva/the-categorical-encoder-you-choose-might-impact-the-model-performance-3dcc2727abb9?source=rss-5da4fd61571c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dcc2727abb9</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-mining]]></category>
            <dc:creator><![CDATA[Cainã Max Couto da Silva]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 17:32:49 GMT</pubDate>
            <atom:updated>2024-04-04T18:10:25.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4hyHs5Etsexok7i4TyiiqA.png" /></figure><p>In this article, I will show you the impact of using specific categorical encoders according to the kind of machine learning model.</p><p>Specifically, I’ll discuss and illustrate how and why the scikit-learn ordinal encoder can harm linear-based models and still be successfully applied to tree-based models.</p><p>Are you ready? Let’s go!</p><h3>Introduction</h3><p>Machine learning models work with numerical inputs. Therefore, to train machine learning models, we must encode categorical variables into a numerical representation so they can work properly.</p><p>Scikit-learn is a powerful open-source library widely used worldwide for machine learning applications. Among its functionalities, it provides estimators (models) and preprocessors to transform the features that will be used as input to the model.</p><p>According to my professional and teaching data science experience, these are the two most used preprocessors to encode categorical features:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder">One-hot encoder</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html#sklearn.preprocessing.OrdinalEncoder">Ordinal encoder</a></li></ul><p>Therefore, I’ll discuss and analyze them for distinct machine-learning models. I will also include the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.TargetEncoder.html#sklearn.preprocessing.TargetEncoder">TargetEncoder</a>, which was recently implemented in scikit-learn (version ≥ 4.1).</p><blockquote>Note: I’ll not elaborate on how these encoders work since there are plenty of materials about it already.</blockquote><p>👉 What about Label Encoder?</p><p>Since we’re talking about scikit-learn, I must say: don’t use it in your input features! The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html">LabelEncoder</a> class provided in the scikit-learn library is suited to encode the <strong>target</strong> variable.</p><p>It’s a common mistake I’ve seen in many notebooks. While it converts (encodes) text to numbers, it cannot transform multiple input features within a pipeline because it was designed to transform the target variable, not the input variables.</p><blockquote>Don’t use LabelEncoder in your input features since it was designed to be applied to the <strong>target </strong>variable, not the features.</blockquote><p>Oh, don’t you use pipelines? Well, perhaps you should! It’s the best way to abstract code complexities, avoid data leakage, and facilitate deployment.</p><p>But let’s continue… We can use <em>OrdinalEncoder </em>instead of the <em>LabelEncoder</em>. <em>OrdinalEncoder </em>encodes categories into ordinal numeric features in the same order they appear in the data. However, this order usually doesn’t make any sense!</p><h3>But… here is the catch:</h3><blockquote>Tree-based models don’t care about the order of the numeric representations provided by OrdinalEncoder, while linear-based models do!</blockquote><p>Consider this note from the course “Machine learning in Python with scikit-learn” developed by the own scikit-learn core contributors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*3dOV7BocIHmu_wCaJCa3qA.png" /><figcaption>Screenshot taken from the course “Machine learning in Python with scikit-learn” — Inria.</figcaption></figure><p>Just think about it: <strong>linear models assume… guess what? Linearity!</strong> Therefore, they consider 5 to be higher than 4, which is higher than 3, and so on. The same applies to distance-based models like KNN, where order matters. Consequently, mapping random numbers to categories will impact the performance of these models.</p><p>On the other hand, tree-based models work by finding the best split based on criteria like entropy/gini (for classification) and squared errors (for regression). Let’s focus here on classification: the split should increase the “purity” of the resulting leaves, so there is no linear assumption here.</p><p>Let’s consider, for example, a category mapped to 1 and another mapped to 5. The model finds the best split to be &gt; 1. It means that separating the data between two groups (0 and 1, or greater than 1) helps the model provide more pure sub-groups.</p><p>A concern might arise when talking about <strong>high cardinality:</strong></p><p>What if we want to use a linear or distance-based model with too many categories in the categorical features?</p><p>When using the one-hot encoder with many categories, too many columns would be created, generating a high computational cost. One possible solution is grouping infrequent categories, and that’s easily achieved using the <em>min_frequency </em>parameter from the <em>OneHotEncoder</em> scikit-learn class. This parameter also exists for the <em>OrdinalEncoder</em>.</p><p>When using TargetEncoder, there is a high chance that an unreliable mean will be computed for the infrequent categories. An option would be to group infrequent categories beforehand or play with the smoothing parameter.</p><p>That said, let’s visually see the impact of the preprocessing step in two models: logistic regression, representing the linear model family, and the Random Forest algorithm as a tree-based model.</p><h3>Comparing the impact of categorical encoders on the model performances</h3><p>For this practical demo, I will use two datasets:</p><ul><li><a href="https://www.openml.org/search?type=data&amp;sort=version&amp;status=any&amp;order=asc&amp;exact_name=adult&amp;id=179">Adult census by OpenML</a></li><li><a href="https://www.kaggle.com/datasets/yeanzc/telco-customer-churn-ibm-dataset">Telco Churn by IBM</a></li></ul><p><strong>Some notes:</strong></p><p>→ I selected only the categorical columns from each dataset to focus on the impact of the categorical encoders.</p><p>→ I selected only two models: Logistic Regression, and Random Forest, representing linear and tree-based models, respectively.</p><p>→ I used cross-validation with the default 5 K-folds to compute five metrics: accuracy, balanced accuracy, precision, recall, and AUC-ROC.</p><p>→ I haven’t performed hyperparameter tuning since the goal was to analyze the influence of the categorical encoders. Default settings were applied.</p><blockquote>I am providing the code notebook <a href="https://github.com/cmcouto-silva/posts/blob/main/notebooks/categorical_encoders_models.ipynb"><strong>here</strong></a>. I’m not placing the code in this article so we can focus on the results, <strong>but feel free to see it</strong>! I made it compact and with comments :)</blockquote><p><strong>Results:</strong></p><p>As a result, that’s the average and standard deviation for each combination of dataset + model + categorical encoder (preprocessor):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4TmUa74_7KW_qF-0wqC0w.png" /><figcaption>Aggregated CV results for each combination of dataset-encoder-model</figcaption></figure><p>To better inspect it, let’s visualize the results split by dataset.</p><p>📊 Adult Census’ dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMS38x0FYYHwOiYNTEvTcA.png" /><figcaption>Dataset: Adult census — Cross-validation results according to the applied encoder.</figcaption></figure><p>The <em>OrdinalEncoder</em>’s performance for the Logistic Regression in the Adult Census dataset was quite poor compared to the other encoders. However, we don’t see a bad performance of the <em>OrdinalEncoder </em>in the Random Forest model!</p><p>Why? Because it doesn’t care about order!<br>It cares about the best splits to make pure leaves.</p><blockquote>Note: I’ve also tested using the option class_weight=’balanced’ for both models (data not shown). The OrdinalEncoder’s impact decreases, but it’s still poorer than the OneHotEncoder and TargetEncoder.</blockquote><p>📊 Telco churn’s dataset</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TakfEUmVioDY-XbP9NoY8g.png" /><figcaption>Dataset: Telco churn — Cross-validation results according to the applied encoder.</figcaption></figure><p>Interestingly, we don’t see the same poor performance as the adult census dataset. However, after an investigation, I found the possible reason: most categorical features already have up to 2–3 unique categories. When only two categories are present, the <em>OrdinalEncoder</em> output ends up looking like the <em>OneHotEncoder</em> dummy’s output.</p><p>In comparison, the adult census has features with many categories (up to 42 categories for some of them). Therefore, here’s another factor to consider: <strong>the cardinality of the categorical features.</strong></p><p>To further demonstrate how a tree-based model works depending on the categorical encoder, let’s analyze a single decision tree for a single categorical column: “marital-status” (the one most important to the random forest model).</p><p><strong>One-hot encoding: </strong>For each split, a tree trained with data processed using the one-hot encoder checks whether a category exists (a condition like “category &lt;= 0.5” means 0 or the absence of the feature).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Yuug3UAHpmf7OkMavUfVg.png" /><figcaption>Decision tree with one-hot encoding applied to a single column</figcaption></figure><p><strong>Ordinal encoding:</strong> ordinal encoding maps discrete integers to the categories. Here, we can see the arbitrary mapping for each category. In this scenario, a condition like “marital-status ≤2.5” means that the marital status is either “Divorced” (0), “Married-AF-spouse” (1), and “Married-civ-spouse” (2), or the remaining possibilities (&gt; 2.5):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*sRbqhPSYYr1I00PvvjmDew.png" /><figcaption>Ordinal encoding mapper for the column “marial-status”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_xa2h3C8SfLmmz5EgPlfNQ.png" /><figcaption>Decision tree with ordinal encoding applied to a single column</figcaption></figure><p>Look again into the tree trained with one-hot encoded data. You’ll see that the model predicts a positive predicted class when “Married-civ-spouse” and “Married-AF-spouse” are True (we’re negating the absence of the condition: absense/0 = False).</p><p>Now, take a closer look at the tree trained with ordinal encoding. By the end, it’s making the same rule decision! For instance:</p><p>The model predicts a negative class for all categories (&gt;2.5) other than “Divorced” (0), “Married-AF-spouse” (1), and “Married-civ-spouse” (2) (right side of the tree). On the left side, we can see that — only when the marital status is &lt;= 0.5 (meaning 0, or “Divorced”) — does the model predict a negative class. Therefore, the model predicts positive results for both “Married-civ-spouse” and “Married-AF-spouse” as well!</p><h3>Final considerations</h3><p>I confirmed what I learned in Inria’s machine learning course: <em>OrdinalEncoder </em>works well with tree-based models but not linear models.</p><p>It’s expected due to the nature of these models — how they work to come up with the final equation/rules).</p><p>Many students complain about linear models without considering their linearity in the preprocessing step. Poor performance usually results from poor data preprocessing. Therefore, <strong>it’s essential to understand how the models and preprocessors work.</strong></p><p>Finally, this work could be expanded to analyze more datasets. Someone could select a dataset with more categorical features and cardinality. Computational costs could also be assessed in this context.</p><p>Again, please take a look at the <a href="https://github.com/cmcouto-silva/posts/blob/main/notebooks/categorical_encoders_models.ipynb">code notebook</a>!</p><p>Any feedback will be well appreciated!</p><h3>References</h3><ul><li><a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing">Scikit-learn preprocessing documentation</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/preprocessing/plot_target_encoder.html#sphx-glr-auto-examples-preprocessing-plot-target-encoder-py">Scikit-learn encoder comparison for a regression task</a></li><li><a href="https://www.fun-mooc.fr/en/courses/machine-learning-python-scikit-learn/">“Machine learning in Python with scikit-learn” course from Inria — developed by the scikit-learn creators and maintainers, and freely available online.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dcc2727abb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>