<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jesus Blanco on Medium]]></title>
        <description><![CDATA[Stories by Jesus Blanco on Medium]]></description>
        <link>https://medium.com/@jesus.blanco_24763?source=rss-578ea73b63d1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*j3YuLfpP1JlBQO6X</url>
            <title>Stories by Jesus Blanco on Medium</title>
            <link>https://medium.com/@jesus.blanco_24763?source=rss-578ea73b63d1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:44:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jesus.blanco_24763/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stop adding machine learning techniques blindly]]></title>
            <link>https://medium.com/sdg-group/stop-adding-machine-learning-techniques-blindly-209b5bf99fe8?source=rss-578ea73b63d1------2</link>
            <guid isPermaLink="false">https://medium.com/p/209b5bf99fe8</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning-engineer]]></category>
            <dc:creator><![CDATA[Jesus Blanco]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 13:21:02 GMT</pubDate>
            <atom:updated>2026-10-13T13:21:02.049Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to achieve the largest performance gain with the least effort</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beGA85qp0SgLI5xhSSG1fg.jpeg" /></figure><p>Anyone who works in <em>machine learning</em> has faced a project where model performance was not good enough. The model worked, just not as well as we needed. And almost immediately, the same question came up:</p><blockquote><strong><em>How can we improve it?</em></strong></blockquote><p>Once we try to answer it, the options pile up quickly: tune hyperparameters, create new features, select existing ones, try different algorithms, build an ensemble, or revisit data cleaning.</p><p>They are all reasonable options. The problem is deciding where to start. Each promises an improvement, but each demands a different combination of time, compute, development effort, and future complexity. Trying them by default can turn model improvement into an endless sequence of experiments.</p><p>I wanted to frame the question differently:</p><blockquote><strong><em>How can I achieve the largest performance gain with the least effort?</em></strong></blockquote><p>More importantly, I wanted to see whether the answer changed depending on what the pipeline already contained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OuFaRhfO72wZJ58Fml6Xw.png" /></figure><h3>I turned the pipeline into a set of switches</h3><p>I used the public <a href="https://www.kaggle.com/competitions/home-credit-default-risk/overview/description">Home Credit Default Risk Kaggle competition</a> as a tabular-data laboratory. Its labeled dataset contains <strong>307,511 loan applications</strong>, and the task is to rank applicants by their risk of repayment difficulties. Credit risk is the case study, but the same question applies to churn, fraud, demand forecasting, or predictive maintenance.</p><p>The Kaggle competition evaluates models with ROC AUC. I report the equivalent normalized Gini, calculated as 2 × AUC − 1, because both metrics measure how well a model ranks higher-risk applicants above lower-risk ones without choosing a classification threshold. This is useful here: only about <strong>8.1%</strong> of applications carry the positive label, so accuracy would be dominated by the majority class and by the chosen cutoff.</p><p>To make differences easier to read, I multiply Gini by 100 throughout the article: <strong>0 points represents random ranking, 100 represents perfect ranking, and one Gini point equals 0.01 on the normalized scale</strong>. Gini measures discrimination; it does not measure probability calibration, financial value, or the quality of a final lending decision.</p><p>Every comparison used the same fixed stratified data splits: 80% for development and 20% for the fixed evaluation sample. I compared pipeline configurations over three fixed development folds, and any preprocessing that learned from the data was fitted only inside the corresponding training fold. The main results score the same applicants in the evaluation sample, so each reported difference changes the named intervention rather than the population being evaluated.</p><p>I organized the experiment around five tests:</p><ul><li><strong>Test 1: add information.</strong> Add prepared external signals and related historical data.</li><li><strong>Test 2: measure conditional returns.</strong> Measure whether the same information block delivers the same gain from different starting points.</li><li><strong>Test 3: turn on feature engineering.</strong> Build deliberate features from the available information.</li><li><strong>Test 4: change the model.</strong> Repeat the comparison across model families.</li><li><strong>Test 5: tune the hyperparameters.</strong> Run hyperparameter optimization (HPO) on the strongest candidates.</li></ul><p>Rather than testing each technique once and in isolation, I compared pipeline configurations in which the main decisions could be switched on or off: external scores, six related historical tables exposed through basic aggregations, and four blocks of advanced <em>feature engineering</em>. I then repeated the comparisons with logistic regression, Random Forest, LightGBM, and CatBoost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCxPbn_XcGlUkgzIpEmZ8A.png" /></figure><p>This allowed me to measure not only whether a technique helped, but how much it added over the immediately preceding pipeline and how its effect changed when combined with other layers. In total, the study comprised <strong>3,820 model fits</strong> across the pipeline comparisons and optimization searches.</p><p>HPO played a different role. It was not crossed with every previous combination: I applied it at the end to three advanced candidates. That distinction matters, and I will return to it later.</p><h3>Test 1: add information</h3><p>I started with a basic CatBoost model using one row per current loan application and only the fields from the main application table, excluding the external scores and linked histories. It achieved <strong>41.1 Gini points</strong>.</p><p>Home Credit provides three numerical columns described only as normalized scores from external data sources. Their origin and construction are not disclosed, so they cannot be identified responsibly as three specific products or databases. I treated them as one prepared information block. With the switch off, all three columns were excluded. With it on, I added their values exactly as provided to the application table. I created no ratios, aggregations, or interactions from them, and kept the base variables, model, parameters, and data split unchanged.</p><p>In a real credit-risk system, analogous enrichment may come from reports or scores supplied by <a href="https://files.consumerfinance.gov/f/201502_cfpb_report_consumer-voices-on-credit-reports-and-scores.pdf">credit bureaus, as documented by the CFPB</a>, which can summarize credit accounts, outstanding debt, and payment history. That is business context, not a claim about the undisclosed origin of these three columns.</p><p>That single on/off change increased performance to <strong>52.1 Gini points</strong>, a gain of <strong>11.0 points</strong>. It was by far the largest individual jump along the trajectory.</p><p>The other way to add information was through related history: previous applications, payments, installments, balances, and other past interactions. I converted each many-row history into one row per applicant using low-effort summaries such as record counts, means, max, and sums. Without the external scores, exposing those six tables through these basic aggregations added <strong>7.8 Gini points</strong> over the basic model.</p><p>But “adding information” does not mean joining every available table indiscriminately. I evaluated the historical sources through a controlled pipeline ablation. Starting from the same reference pipeline containing the external scores, I added one historical table at a time, reduced it to basic applicant-level aggregations, and trained all four model families over the same three development folds. In the chart, each point is the mean Gini gain across families and each line is the observed range. The external-score row uses the base application data as its reference; the historical rows use the external-score pipeline.</p><p>This is a pipeline-level ablation: it measures the contribution of adding an entire source, not the importance of individual variables inside a fitted model. The comparisons showed important differences: point-of-sale and cash-loan history and installment history were among the most valuable, while detailed monthly credit-bureau balances contributed approximately zero depending on the model family.</p><p>Because the chart summarizes development-fold effects across four families, its external-score average is not the same quantity as the <strong>11.0-point</strong> CatBoost result on the fixed evaluation sample. The axis is also interrupted so the smaller historical gains remain visible beside the much larger external-score effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yw3oAo_GnaSyeLv7UIdCrA.png" /></figure><p>This was the first lesson about effort. Testing a source through a simple representation can be cheap and can quickly reveal whether it contains useful signal. Keeping it forever is not cheap: every table adds contracts, joins, quality controls, leakage risk, and monitoring. A source should earn its place through marginal contribution, not merely through availability.</p><h3>Test 2: measure conditional returns</h3><p>The switches also made it possible to ask a more precise question: does the same intervention buy the same improvement from every starting point?</p><p>With CatBoost on the same evaluation sample, the four relevant states were: <strong>41.1 points</strong> for base data, <strong>48.9</strong> for history only, <strong>52.1</strong> for external scores only, and <strong>55.0</strong> when both information blocks were present.</p><p>Now the conditional gains are easy to see. Historical data added <strong>7.8 points</strong> without external scores, but <strong>2.9 points</strong> once they were present. Read in the other direction, the external scores added <strong>11.0 points</strong> without history and <strong>6.1 points</strong> once history was already available.</p><p>This does not prove that one source causes the other to work less, nor does it identify the internal mechanism. It shows that their marginal contributions are conditional on the pipeline state. Some redundancy is a plausible explanation, but the experiment only establishes the changing gain, not its cause.</p><h3>Test 3: turn on feature engineering</h3><p>I then examined what happened when the available information was represented more effectively. On top of the basic aggregations, I built <strong>67 deliberate features</strong> covering application ratios, payment behavior, recency, trends, volatility, and consistency across sources.</p><p><em>Feature engineering</em> helped in both contexts, but not by the same amount.</p><p>Without external scores, it added <strong>3.3 Gini points</strong>. Once those scores were present, it added <strong>1.9 points</strong>. It still helped, but its measured marginal contribution was smaller from the richer starting point. The comparison establishes that change in contribution; it does not identify which relationships the model had already captured.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1qEjOMUtFhBn-e4eZIjvA.png" /></figure><p>This difference puts its cost into perspective. Creating advanced features is not just a matter of writing divisions and aggregations. It requires forming hypotheses, defining time windows, preventing leakage, handling edge cases, validating transformations, and governing them in production. That work can pay off when a basic representation hides important relationships. But we should not assume it will produce the same gain at every stage of a project.</p><p>Combining this test with the previous one revealed a particularly interesting trade-off: buying or reusing a ready-made signal versus constructing it from our own history.</p><p>The route using only the three external scores achieved <strong>52.1 Gini points</strong>. The route using six historical tables and all 67 advanced features, but no external score, reached <strong>52.2 points</strong>.</p><p>The point estimates are almost identical, but that alone does not prove the two routes statistically equivalent. The implementation burden is also very different. One route depends on three prepared signals that may be subject to price, permissions, and coverage. The other requires six tables, 262 final features, and considerably more design and maintenance logic. The more attractive route depends on each organization’s actual constraints.</p><h3>Test 4: change the model</h3><p>Repeating the configurations across four model families ensured that the conclusion did not depend entirely on a CatBoost quirk. The exact magnitude changed, but the main families showed the same general pattern: information layers produced the large gains, and <em>feature engineering</em> added a later increment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-omYkFkyucEA8C7ARS04_g.png" /></figure><p>Using the same complete information before HPO, CatBoost led development with <strong>56.5 Gini points</strong>. LightGBM was close at <strong>56.1 points</strong>, while its observed pipeline runtime was roughly three times shorter: 16.2 seconds per fold instead of 45.5.</p><p>These are observed pipeline times, including preprocessing, training, and prediction, rather than intrinsic properties of each model family. Different implementations or hardware could change the ranking.</p><p>Why did some longer-running pipelines perform worse? Runtime pays for operations, not automatically for useful signal. In this setup, logistic regression expanded categorical variables before fitting the model, while Random Forest built 200 decision trees. The two boosting implementations processed the same table more efficiently: LightGBM was fastest, while CatBoost spent more time handling categories and gained only 0.4 Gini points over it. These details explain the observed pipelines, not a universal ranking of algorithms.</p><p>Even model choice therefore has no context-independent winner: a few tenths of a Gini point, retraining frequency, and compute budget can all change the decision.</p><p>The goal was not to build another performance ranking. It was to test whether the conclusions about the interventions survived a change of model family.</p><h3>Test 5: tune the hyperparameters</h3><p>Only then did I optimize three candidates selected from development: the complete CatBoost and LightGBM pipelines, plus a reduced CatBoost pipeline without the temporal and cross-source feature blocks. For each candidate, I compared random search with Optuna’s TPE search. Random search samples configurations independently; TPE uses previous trials to guide later choices under the same time budget. TPE produced the selected configuration for all three finalists.</p><p>The untuned complete CatBoost pipeline averaged <strong>56.5 points</strong> across development folds and scored <strong>56.8</strong> on the fixed evaluation sample. HPO raised that evaluation result to <strong>57.8 Gini points</strong>, a gain of <strong>0.92 points</strong>. It added <strong>0.79 points</strong> to complete LightGBM and <strong>0.60 points</strong> to reduced CatBoost. All three increments were positive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kuOXVI_0kET887QsTv6lqg.png" /></figure><p>On the best pipeline, a <strong>0.92-point</strong> gain can be highly valuable in a mature system where a few tenths of a point affect thousands of decisions. Across all three finalists and both search strategies, the optimization study required 1,129 trials and almost 19 aggregate fit-hours, calculated as the sum of the individual training times rather than elapsed wall-clock time.</p><p>HPO worked. What it did not do was deliver a gain of the same magnitude as adding information or building a better representation. In this experiment, it was a last-mile investment.</p><p>I only tuned advanced finalists, so this test measures HPO at the end of the sequence, not its effect on a basic pipeline.</p><h3>Which intervention bought the largest gain with the least effort?</h3><p>After working through the tests, the complete trajectory can be summarized as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V0qq63PAJwl7ucI2iQlRrA.png" /></figure><p>External scores added <strong>11.0 points</strong>; related history added another <strong>2.9 points</strong>; <em>feature engineering</em>, <strong>1.9 points</strong>; and HPO, <strong>0.9 points</strong>. These gains belong to this experiment, but their progression gives a clear answer to the question that started it.</p><p>The largest improvement came from adding information that could contain new signal. Here, three ready-made external scores transformed the baseline more than any later intervention. When a model underperforms, the first move should therefore be to expose promising new sources, histories, or signals through the simplest reliable representation and test whether they change what the model can learn.</p><p>Once that information was available, <em>feature engineering</em> produced the next meaningful gain. It also demanded much more deliberate work: understanding the data, imagining useful variables, choosing aggregations and interactions, handling edge cases, and maintaining the resulting logic. It was valuable because it extracted more from information the pipeline already contained.</p><p>HPO came last. Modern libraries make an optimization study relatively easy to configure, but the search itself consumes time and compute. It improved every finalist by less than one Gini point. HPO was the icing on the cake: useful for polishing a strong pipeline, but too small a lever to compensate for missing information.</p><h3><strong>Conclusion</strong></h3><p>That suggests a simple order of effort:</p><ol><li><strong>Add information.</strong> Look for new sources, variables, histories, or prepared signals that could change what the model is able to learn.</li><li><strong>Curate that information.</strong> Use targeted feature engineering to expose useful relationships, behaviors, and temporal patterns.</li><li><strong>Tune the strongest candidates.</strong> Spend the optimization budget only after the data and its representation are already competitive.</li></ol><p>The exact costs will vary: external data can involve acquisition and governance, historical tables need reliable pipelines, engineered features create maintenance work, and HPO consumes compute. This single tabular case is not a universal proof. What it shows is that marginal returns depend on what the pipeline already contains, so techniques should not be treated as equally promising at every stage.</p><p>The next time a model underperforms, do not start by polishing it. <strong>First add information, then extract more value from it, and only then optimize the remaining details.</strong> At every stage, test what the intervention actually bought before paying for the next one.</p><h3>Who am I</h3><p>Jesus Blanco is a Senior Consultant in SDG Group’s AI practice and an AI/ML Engineer with a background in computer engineering and mathematics. He has worked across data engineering, data science, and machine learning, delivering end-to-end solutions from data preparation to production deployment. His experience spans banking, automotive, and consulting, and he enjoys applying machine learning to practical problems, improving processes, and turning data into useful decisions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=209b5bf99fe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sdg-group/stop-adding-machine-learning-techniques-blindly-209b5bf99fe8">Stop adding machine learning techniques blindly</a> was originally published in <a href="https://medium.com/sdg-group">SDG Group</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>