<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Navya Illa on Medium]]></title>
        <description><![CDATA[Stories by Navya Illa on Medium]]></description>
        <link>https://medium.com/@9navya9?source=rss-fa313d1ceba7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1r2moVUiMMZxERQw3-nxgA.jpeg</url>
            <title>Stories by Navya Illa on Medium</title>
            <link>https://medium.com/@9navya9?source=rss-fa313d1ceba7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:20:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@9navya9/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Predicting Bank Marketing Responses Using the SEMMA Methodology]]></title>
            <link>https://medium.com/@9navya9/predicting-bank-marketing-responses-using-the-semma-methodology-5dad2c2d1690?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dad2c2d1690</guid>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[marketing-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 09:18:52 GMT</pubDate>
            <atom:updated>2026-11-09T09:18:52.818Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Banks often contact customers to promote financial products such as term deposits. However, contacting every customer can be expensive and inefficient.</p><p>A predictive model can help the bank identify customers who are more likely to subscribe before starting a campaign. This allows marketing teams to focus their time and resources on the most promising customers.</p><p>In this project, I applied the SEMMA methodology to the Bank Marketing dataset. The goal was to predict whether a customer would subscribe to a term deposit after a marketing campaign.</p><p>SEMMA represents five stages:</p><ol><li>Sample</li><li>Explore</li><li>Modify</li><li>Model</li><li>Assess</li></ol><h3>Dataset</h3><p>The dataset used in this project comes from the UCI Machine Learning Repository.</p><p>It contains information about customers and their previous interactions with a bank. Important variables include:</p><ul><li>Age</li><li>Job</li><li>Marital status</li><li>Education</li><li>Account balance</li><li>Housing loan status</li><li>Personal loan status</li><li>Contact method</li><li>Previous campaign information</li></ul><p>The target variable indicates whether the customer subscribed to a term deposit.</p><p>This is a binary-classification problem because the outcome is either subscription or no subscription.</p><h3>1. Sample</h3><p>The first SEMMA stage is Sample.</p><p>In this stage, I defined the population used for the marketing analysis and divided the data into training and testing sets.</p><p>A stratified split was used so that the proportion of customers who subscribed and did not subscribe remained similar in both datasets.</p><p>This is important because the target classes are not evenly balanced. Most customers did not subscribe, while only a smaller percentage accepted the term-deposit offer.</p><p>A reproducible split was used so that the analysis could be repeated consistently.</p><h3>2. Explore</h3><p>The second stage is Explore.</p><p>I examined the data types, target distribution, unknown values, and distributions of important features.</p><p>Several categorical columns contained values such as “unknown.” These values were reviewed carefully instead of automatically being treated as missing values because they may represent a meaningful category.</p><p>I also examined the target distribution to understand the imbalance between subscribers and non-subscribers.</p><p>Because of this imbalance, accuracy alone would not provide a complete evaluation of the model. Metrics such as ROC-AUC, PR-AUC, precision, recall, and campaign-capacity recall were also considered.</p><h3>Preventing Data Leakage</h3><p>One important modeling decision involved the duration feature.</p><p>Duration represents how long the customer was contacted during the campaign. However, this value is only known after the contact has already taken place.</p><p>The goal of this project is to predict whether a customer will subscribe before contacting them. Therefore, duration was excluded from the model.</p><p>Including duration would create data leakage because the model would use information that would not be available when making a real pre-contact prediction.</p><p>This decision was documented in the project’s leakage-decision file.</p><h3>3. Modify</h3><p>The third SEMMA stage is Modify.</p><p>In this stage, the data was prepared for machine learning.</p><p>Numerical features were imputed and scaled when necessary. Categorical features were encoded so that the algorithms could process them.</p><p>All preprocessing operations were placed inside a scikit-learn pipeline.</p><p>This prevents data leakage because the transformations are learned only from the training data and then applied to the testing data.</p><p>Using a pipeline also makes the final model easier to reproduce and deploy.</p><h3>4. Model</h3><p>The fourth stage is Model.</p><p>I compared three classification algorithms:</p><ul><li>Logistic regression</li><li>Random forest</li><li>Histogram gradient boosting</li></ul><p>Logistic regression provides a simple and interpretable baseline.</p><p>Random forest uses multiple decision trees and can model nonlinear relationships.</p><p>Histogram gradient boosting builds models sequentially and is effective for learning complex patterns in structured data.</p><p>The models were compared using stratified cross-validation. After evaluating their results, histogram gradient boosting was selected as the final model because it achieved the strongest overall performance.</p><h3>5. Assess</h3><p>The fifth and final SEMMA stage is Assess.</p><p>The selected model achieved a held-out ROC-AUC of approximately <strong>0.7927</strong> and a PR-AUC of approximately <strong>0.4512</strong>.</p><p>ROC-AUC measures how well the model ranks customers who are likely to subscribe above customers who are unlikely to subscribe.</p><p>PR-AUC is useful when the positive class is smaller because it focuses on the relationship between precision and recall for potential subscribers.</p><p>The project also evaluated a campaign-capacity threshold. Instead of contacting every customer, the bank can contact the 20 percent of customers with the highest predicted probabilities.</p><p>At a 20-percent campaign capacity, the model achieved a recall of approximately <strong>0.6087</strong>.</p><p>This means the bank can identify a meaningful portion of potential subscribers while contacting only a limited group of customers.</p><p>The classification report, ROC curve, and precision-recall curve provide additional insight into the model’s performance.</p><h3>Business Applications</h3><p>The model can support several marketing activities:</p><ul><li>Prioritizing customers for campaign outreach</li><li>Reducing unnecessary calls</li><li>Improving campaign efficiency</li><li>Identifying customers with higher subscription potential</li><li>Allocating marketing resources more effectively</li></ul><p>Customers with high predicted probabilities can be contacted first, while customers with lower probabilities can be placed in a lower-priority group.</p><p>The model should support marketing decisions rather than completely replace human judgment.</p><h3>SAS Enterprise Miner Mapping</h3><p>The project also includes a SAS Enterprise Miner node mapping.</p><p>This mapping demonstrates how the SEMMA workflow can be represented in a visual data-mining environment.</p><p>The stages can be connected to activities such as sampling data, exploring distributions, modifying variables, training models, and assessing model performance.</p><p>Screenshots should be added after personally running the flow in SAS Enterprise Miner.</p><h3>Limitations</h3><p>The model has several limitations.</p><p>Customer behavior may change based on market conditions, interest rates, economic factors, and the timing of the campaign.</p><p>The model also depends on the quality and completeness of the customer information.</p><p>In addition, a high predicted probability does not guarantee that a customer will subscribe.</p><p>The model should be monitored regularly and retrained when new campaign data becomes available.</p><p>Future improvements could include probability calibration, additional customer information, fairness analysis, and testing the model on newer campaigns.</p><h3>Conclusion</h3><p>This project applied all five stages of the SEMMA methodology to the Bank Marketing dataset.</p><p>The project sampled the campaign population, explored the data, modified the features using a leakage-safe pipeline, compared multiple models, and assessed the final model using appropriate metrics.</p><p>Histogram gradient boosting was selected as the final model, achieving a held-out ROC-AUC of approximately 0.7927 and a PR-AUC of approximately 0.4512.</p><p>At a 20-percent campaign capacity, the model achieved a recall of approximately 0.6087.</p><p>The project demonstrates how machine learning can help banks improve marketing efficiency by prioritizing customers who are most likely to subscribe to a term deposit.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dad2c2d1690" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discovering Customers at Risk of Churning Using CRISP-DM]]></title>
            <link>https://medium.com/@9navya9/discovering-customers-at-risk-of-churning-using-crisp-dm-4d5ff67924c9?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d5ff67924c9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-churn]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 09:15:12 GMT</pubDate>
            <atom:updated>2026-11-09T09:15:12.942Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Customer churn is one of the biggest challenges in the telecommunications industry. When customers leave, companies lose recurring revenue and must spend additional money to acquire replacements.</p><p>A data-driven churn-prediction system can help a company identify customers who are likely to leave and prioritize them for retention outreach.</p><p>In this project, I applied the CRISP-DM methodology to the IBM Telco Customer Churn dataset. The goal was to build a machine-learning pipeline that predicts customer churn and supports targeted retention decisions.</p><p>The project follows the six CRISP-DM phases:</p><ol><li>Business Understanding</li><li>Data Understanding</li><li>Data Preparation</li><li>Modeling</li><li>Evaluation</li><li>Deployment</li></ol><h3>Business Understanding</h3><p>The business problem is to identify customers who have a high probability of churning.</p><p>The company cannot contact every customer with a retention offer because outreach resources are limited. Therefore, the model should help prioritize customers who are most likely to leave.</p><p>The main business objective is to maximize the number of potential churners identified within a limited outreach capacity.</p><p>The prediction system should also be reproducible, interpretable, and suitable for future deployment.</p><h3>Dataset</h3><p>The project uses the IBM Telco Customer Churn dataset.</p><p>The dataset contains customer information such as:</p><ul><li>Gender</li><li>Senior-citizen status</li><li>Partner and dependent status</li><li>Tenure</li><li>Phone and internet services</li><li>Contract type</li><li>Payment method</li><li>Monthly charges</li><li>Total charges</li><li>Churn status</li></ul><p>The target variable is Churn, which indicates whether a customer left the company.</p><p>Because the target has two possible outcomes, this is a binary-classification problem.</p><h3>Data Understanding</h3><p>The first step was to examine the structure and quality of the dataset.</p><p>I reviewed the column names, data types, missing values, and the distribution of the target variable.</p><p>One important data-quality issue was found in the TotalCharges column. Although this column should contain numerical values, some records contained blank strings.</p><p>These values were converted to numeric format, and invalid entries were handled before modeling.</p><p>The target distribution also showed that there were more customers who did not churn than customers who churned. Because of this class imbalance, accuracy alone would not be a reliable evaluation metric.</p><p>Metrics such as ROC-AUC, PR-AUC, precision, recall, and the confusion matrix were also used.</p><h3>Data Preparation</h3><p>The customer identifier was removed because it does not provide meaningful predictive information.</p><p>The numerical and categorical features were processed separately. Numerical features were prepared for modeling, while categorical features were encoded into a machine-readable format.</p><p>The dataset was divided into training and testing sets using a stratified split. Stratification helped maintain a similar proportion of churned and non-churned customers in both sets.</p><p>All preprocessing steps were placed inside a scikit-learn pipeline.</p><p>This is important because it prevents data leakage. The transformations are learned only from the training data and then applied to the test data.</p><p>As a result, the evaluation better represents how the model would perform on new, unseen customers.</p><h3>Modeling</h3><p>Two machine-learning models were compared:</p><ul><li>Logistic regression</li><li>Random forest</li></ul><p>Logistic regression provides a simple and interpretable baseline. It estimates the probability that a customer will churn based on the available features.</p><p>Random forest is an ensemble model that combines multiple decision trees. It can capture nonlinear relationships and interactions between features.</p><p>Both models were evaluated using stratified cross-validation.</p><p>After comparing their performance, logistic regression was selected as the final model. It provided strong predictive performance while remaining relatively easy to interpret and deploy.</p><h3>Evaluation</h3><p>The final model achieved a held-out ROC-AUC of approximately <strong>0.8416</strong>.</p><p>The PR-AUC was approximately <strong>0.6337</strong>.</p><p>ROC-AUC measures how well the model separates customers who churn from customers who do not churn.</p><p>PR-AUC is especially useful when the positive class is smaller because it focuses on the relationship between precision and recall for churned customers.</p><p>The project also used a top-30-percent outreach policy. Under this policy, the company contacts the 30 percent of customers with the highest predicted churn probabilities.</p><p>Using this threshold, the model achieved an outreach recall of approximately <strong>0.6765</strong>.</p><p>This means the company can identify a meaningful portion of likely churners while contacting only a limited percentage of customers.</p><p>The confusion matrix and classification report provide additional information about correct predictions, false positives, and false negatives.</p><p>In a retention campaign, false negatives can be costly because they represent customers who may churn but are not contacted. However, contacting too many low-risk customers can also waste resources.</p><p>Therefore, the prediction threshold should be selected based on the company’s outreach capacity and the relative cost of different errors.</p><h3>Deployment</h3><p>The final preprocessing and modeling pipeline was saved as a serialized model file.</p><p>A scoring script was also created for deployment. It accepts information about a single customer, applies the same preprocessing steps used during training, generates a churn probability, and assigns an operational priority label.</p><p>For example, a customer with a high predicted churn probability could be marked as a high-priority retention case.</p><p>This allows the model to support batch scoring or future integration into a customer-retention system.</p><h3>Business Use</h3><p>The model can support several business activities:</p><ul><li>Prioritizing retention calls</li><li>Offering discounts to high-risk customers</li><li>Identifying customers who need additional support</li><li>Designing targeted loyalty campaigns</li><li>Estimating the number of customers at risk of leaving</li></ul><p>The model should be used as a decision-support tool. It should help employees prioritize their work but should not completely replace human judgment.</p><h3>Limitations</h3><p>There are several limitations to this project.</p><p>Customer behavior may change over time because of pricing changes, competitor offers, service quality, or broader market conditions.</p><p>The model should therefore be monitored for changes in data and performance.</p><p>The model also does not explain every reason why a customer may churn. A high-risk prediction indicates increased likelihood, but it does not guarantee that the customer will leave.</p><p>The company should also evaluate whether retention offers are effective and whether the model performs consistently across different customer groups.</p><p>Future improvements could include model calibration, regular retraining, additional customer-service information, and monitoring for data drift.</p><h3>AI-Assisted Review</h3><p>The project also included an AI-assisted critique and revision process.</p><p>The review focused on data quality, leakage prevention, model evaluation, reproducibility, deployment, and business limitations.</p><p>The recommendations were reviewed and incorporated where appropriate. This helped improve the clarity and reliability of the final project.</p><h3>Conclusion</h3><p>This project applied the complete CRISP-DM methodology to the customer-churn problem.</p><p>The workflow began with a business objective and continued through data understanding, data preparation, model comparison, evaluation, and deployment.</p><p>The final logistic-regression pipeline achieved a held-out ROC-AUC of approximately 0.8416 and a PR-AUC of approximately 0.6337.</p><p>Using a top-30-percent outreach policy, the model achieved an outreach recall of approximately 0.6765.</p><p>The project demonstrates how machine learning can help telecommunications companies prioritize retention efforts and make better use of limited outreach resources.</p><p>The most important lesson is that a successful data-science project requires more than choosing an algorithm. It requires a clear business goal, careful data preparation, leakage-safe evaluation, meaningful metrics, and a practical deployment plan.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d5ff67924c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discovering Customer Segments and Product Associations from Online Retail Data]]></title>
            <link>https://medium.com/@9navya9/discovering-customer-segments-and-product-associations-from-online-retail-data-f73cb4072903?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f73cb4072903</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-segmentation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 09:07:24 GMT</pubDate>
            <atom:updated>2026-11-09T09:07:24.347Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Online retail companies collect large amounts of transaction data every day. Each purchase contains information about the customer, products, quantities, prices, and purchase time.</p><p>When analyzed properly, this data can answer important business questions:</p><ul><li>Which customers are the most valuable?</li><li>Which customers purchase frequently?</li><li>Which products are commonly purchased together?</li><li>How can customers be grouped for targeted marketing?</li><li>Which products should be recommended or promoted together?</li></ul><p>In this project, I applied the Knowledge Discovery in Databases, or KDD, process to the Online Retail dataset. The analysis focused on customer segmentation and product association discovery.</p><p>The project used two major data-mining techniques:</p><ol><li>K-means clustering for customer segmentation</li><li>Apriori association-rule mining for product relationships</li></ol><h3>Dataset</h3><p>The dataset used in this project is the Online Retail dataset from the UCI Machine Learning Repository.</p><p>It contains transaction-level information such as:</p><ul><li>Invoice number</li><li>Product description</li><li>Quantity</li><li>Invoice date</li><li>Unit price</li><li>Customer identifier</li><li>Country</li></ul><p>Each row represents a product included in a customer transaction.</p><p>The dataset is useful for analyzing purchasing behavior because it contains both customer-level and product-level information.</p><h3>The KDD Process</h3><p>The project follows five major stages of the KDD process:</p><ol><li>Selection</li><li>Preprocessing</li><li>Transformation</li><li>Data Mining</li><li>Interpretation and Evaluation</li></ol><p>Each stage prepares the data for the next stage and helps convert raw transactions into useful business knowledge.</p><h3>1. Selection</h3><p>The first step was to define the population of transactions and customers used in the analysis.</p><p>The analysis focused on valid retail purchases. Records that did not represent meaningful completed transactions were excluded.</p><p>The project investigated two questions:</p><ol><li>How can customers be grouped according to their purchasing behavior?</li><li>Which products are frequently purchased together?</li></ol><p>These questions require two different data representations. Customer segmentation requires customer-level features, while association mining requires invoice-level product baskets.</p><h3>2. Preprocessing</h3><p>The raw transaction data required several cleaning steps.</p><p>Cancelled invoices were removed because they do not represent completed purchases. Transactions with nonpositive quantities or nonpositive prices were also excluded.</p><p>Rows without customer identifiers were removed from the customer-segmentation analysis because they could not be connected to a specific customer.</p><p>Missing product descriptions were removed, and product names were normalized to reduce duplicate variations in product text.</p><p>After preprocessing, the project contained:</p><ul><li>4,338 customers</li><li>17,138 valid multi-item invoices</li></ul><p>These cleaned transactions formed the basis for the rest of the analysis.</p><h3>3. Transformation</h3><p>The transaction data was transformed into two different formats.</p><h3>RFM Features</h3><p>For customer segmentation, I created RFM features.</p><p>RFM stands for:</p><ul><li>Recency: how recently a customer made a purchase</li><li>Frequency: how often a customer purchased</li><li>Monetary value: how much a customer spent</li></ul><p>These features summarize customer behavior in a way that can be used by a clustering algorithm.</p><p>For example, a customer who purchased recently, purchased many times, and spent a large amount would have strong RFM values.</p><h3>Product Baskets</h3><p>For association-rule mining, each invoice was converted into a product basket.</p><p>A basket contains all the products purchased together in one invoice. This format allows the Apriori algorithm to identify product combinations that frequently appear together.</p><h3>4. Customer Segmentation Using K-Means</h3><p>K-means clustering was used to divide customers into groups with similar purchasing behavior.</p><p>The algorithm assigns customers to clusters based on their RFM features. Customers within the same cluster should have similar behavior, while customers in different clusters should be more distinct.</p><p>To select the number of clusters, I evaluated different values of k using silhouette analysis.</p><p>The silhouette score measures how well customers fit within their assigned cluster compared with other clusters. A higher score generally indicates better separation between groups.</p><p>The analysis selected two clusters as the best solution.</p><p>After creating the clusters, I profiled them using their RFM statistics. This allowed the groups to be interpreted from a business perspective.</p><p>For example, one cluster may represent more active or higher-value customers, while the other may represent customers with lower purchasing activity.</p><p>These segments could support different marketing strategies. High-value customers might receive loyalty rewards, while less-active customers could receive re-engagement offers.</p><h3>5. Product Association Mining</h3><p>The second analysis used Apriori association-rule mining.</p><p>The goal was to identify products that frequently appeared in the same invoice.</p><p>The analysis used three important measures:</p><h3>Support</h3><p>Support measures how frequently a product combination appears in the complete transaction database.</p><p>A higher support value means the combination occurs more often.</p><h3>Confidence</h3><p>Confidence measures how often a product is purchased when another product is purchased.</p><p>For example, if customers who buy Product A frequently also buy Product B, the rule may have high confidence.</p><h3>Lift</h3><p>Lift compares the observed relationship between products with what would be expected if the products were independent.</p><p>A lift greater than one suggests that the products are associated more strongly than expected by chance.</p><p>The analysis identified:</p><ul><li>282 frequent itemsets</li><li>96 filtered association rules</li></ul><p>These rules can provide useful ideas for product recommendations, promotions, cross-selling, and store organization.</p><p>For example, products that are frequently purchased together could be promoted as a bundle or recommended to customers during checkout.</p><h3>Results</h3><p>The final results combined customer segmentation with product-association discovery.</p><p>The clustering analysis produced two customer groups based on recency, frequency, and monetary value.</p><p>The association analysis produced frequent product combinations and filtered rules using support, confidence, and lift.</p><p>Together, these results provide two different perspectives:</p><ul><li>Customer segmentation explains who the customers are.</li><li>Association mining explains what products are purchased together.</li></ul><p>This combination can help a business create more targeted marketing strategies.</p><p>For example, a company could identify a high-value customer segment and then recommend products that are commonly purchased together by similar customers.</p><h3>Business Applications</h3><p>The results can support several business activities.</p><h3>Targeted Marketing</h3><p>Different customer segments can receive different promotions based on their purchasing behavior.</p><h3>Customer Retention</h3><p>Customers with declining activity can be identified and targeted with re-engagement campaigns.</p><h3>Product Recommendations</h3><p>Association rules can be used to recommend products that are frequently purchased together.</p><h3>Bundle Promotions</h3><p>Frequently associated products can be offered as discounts or bundles.</p><h3>Inventory Planning</h3><p>Products that are commonly purchased together may need to be stocked together or monitored as a group.</p><h3>Limitations</h3><p>This analysis has several limitations.</p><p>First, customer behavior depends on the time period represented in the dataset. The discovered segments and product relationships may change over time.</p><p>Second, association rules show relationships but do not prove causation. If two products are purchased together, it does not mean that purchasing one product causes the purchase of the other.</p><p>Third, clustering results depend on feature selection, scaling, and the chosen number of clusters.</p><p>Finally, product descriptions may contain inconsistencies, abbreviations, or variations that can affect association results.</p><p>For these reasons, the results should be reviewed by business experts and validated using newer data before being used in production.</p><h3>Conclusion</h3><p>This project applied the KDD process to the Online Retail dataset to discover customer segments and product relationships.</p><p>The analysis included:</p><ul><li>Selection of valid transactions</li><li>Removal of cancelled and invalid records</li><li>Cleaning of customer and product information</li><li>Creation of RFM customer features</li><li>K-means customer segmentation</li><li>Invoice-basket transformation</li><li>Apriori association-rule mining</li><li>Evaluation and interpretation of the results</li></ul><p>The final analysis identified 4,338 customers, 17,138 valid multi-item invoices, two customer clusters, 282 frequent itemsets, and 96 filtered association rules.</p><p>The project demonstrates how raw transaction data can be transformed into meaningful business knowledge. Customer segmentation can help businesses understand their customers, while association rules can support recommendations, cross-selling, and product-bundle decisions.</p><p>The most important lesson is that data mining is not only about applying algorithms. It also requires careful data selection, preprocessing, transformation, evaluation, and business interpretation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f73cb4072903" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Data-Driven Models Read the Soil Moisture Beneath Our Feet]]></title>
            <link>https://medium.com/@9navya9/how-machine-learning-reads-the-moisture-beneath-our-feet-f3fa2096adaf?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3fa2096adaf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 06:35:14 GMT</pubDate>
            <atom:updated>2026-10-11T06:30:00.782Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLdCb-uJQse-2p6Af7idQw.png" /></figure><p>Soil moisture is invisible from above, yet drought alerts, irrigation schedules, and flood forecasts all depend on it. Two spots in the same field can hold very different amounts of water, and the moisture at a single spot keeps changing over time. A model has to learn from measurements that vary across space, depth, and time, and combine sensors that see the world at different scales.</p><p>In this article, I review a 2026 survey, <em>A Survey on Data-Driven Models for Soil Moisture Regression and Classification</em> by Ilektra Tsimpidi, George Georgoulas, Vidya Sumathy, and George Nikolakopoulos of Luleå University of Technology. I focus on the architecture of the modeling workflow, the five model families, what the studies reveal when they compare against baselines, how the models are measured, and my own assessment of the paper.</p><p>The survey organizes the field into five model families and a single shared pipeline. Its central message is that the choice of model matters less than defining the target, aligning the data, and testing under realistic conditions.</p><p><strong>Why soil moisture is hard to model</strong></p><p>Traditional physics-based models rely on explicit hydrological equations, such as water balance models. They are principled, but they require high-quality inputs and become computationally expensive at large scale.</p><p>Data-driven models take another route by learning the relationship between soil moisture and observed variables directly. That flexibility moves the difficulty into the data and the experimental design. The survey points to four sources of complexity:</p><ol><li>Soil moisture varies across space, time, and soil depth.</li><li>Climate, soil, vegetation, and topography interact in nonlinear ways.</li><li>Satellite products and field sensors measure at very different spatial scales.</li><li>Ground observations are scarce compared with the areas that need predictions.</li></ol><p><strong>The modeling pipeline</strong></p><p>Whatever model is chosen, the survey describes a shared workflow. Data are gathered from sensors and satellites, preprocessed and aligned, turned into features such as weather, soil, and terrain variables, passed to a model, and finally validated.</p><p>Typical inputs include precipitation, temperature, solar radiation, vegetation indices, radar backscatter, soil texture, elevation, slope, and antecedent soil moisture. Common sources include the International Soil Moisture Network, SMAP and SMOS satellites, Sentinel-1 radar, ERA5-Land reanalysis, and MODIS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHbpFgIIxFuwXEVeywvrVg.png" /><figcaption><em>The soil moisture modeling pipeline, redrawn from the survey’s framework</em></figcaption></figure><p>In my view, this pipeline is the most useful idea in the paper. Many weak soil moisture studies fail at the first or last stage, through poorly aligned data or leaky testing, rather than at the model itself.</p><p><strong>Two prediction tasks: regression and classification</strong></p><p>Regression predicts a continuous moisture value and is by far the more common task. The survey describes five settings: point estimation, temporal forecasting, multi-depth prediction, gap filling for incomplete records, and spatial downscaling, where a coarse satellite product is refined to finer resolution.</p><p>Classification assigns moisture to categories such as dry, moderate, and wet. It is less precise, but it works when data are limited and is easier to interpret for drought monitoring and irrigation decisions. The choice between the two is as much a product decision as a modeling one.</p><p><strong>Five families of data-driven models</strong></p><p>The survey organizes models by learning paradigm, from the oldest approaches to the newest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8g7g3arZZbwwU9C91nWBgQ.png" /><figcaption><em>The five model families and representative methods</em></figcaption></figure><p><strong>Statistical time-series models.</strong> ARIMA, SARIMA, and related models are interpretable, robust when data are limited, and able to provide confidence intervals. The cost is that they assume a form for the data-generating process. One reviewed study used SARIMA to forecast soil water content and salinity 48 hours ahead for irrigation with reclaimed water.</p><p><strong>Geostatistical methods.</strong> Kriging, inverse distance weighting, and co-Kriging treat moisture as a spatial process in which nearby points are related, turning scattered measurements into continuous maps. Ordinary Kriging was used to fill gaps in SMAP satellite pixels.</p><p><strong>Classical machine learning.</strong> Random Forests, XGBoost, support vector machines, and neural networks learn nonlinear relationships from mixed inputs without hand-written equations. Notable examples include <strong>SMRFR</strong>, a Random Forest product estimating moisture at five depths, daily and globally, at 9 km resolution for 2000 to 2023, and an XGBoost model tuned with a genetic algorithm that reached <strong>R² = 0.891</strong>.</p><p><strong>Deep learning.</strong> LSTMs, CNNs, graph-based ConvLSTMs, and transformers learn features directly from raw time series, grids, and images. Transformers were competitive for long horizons and deep soil layers, though not always better than LSTMs. A newer trend is physics-guided deep learning, where physical constraints are built into training.</p><p><strong>Probabilistic and Bayesian models.</strong> Hierarchical Bayesian models and Gaussian processes report uncertainty alongside the prediction, which matters when data are sparse or when someone must decide whether to act on a forecast.</p><p><strong>Comparing the families</strong></p><p>The survey describes each family but stops short of a side-by-side comparison. The summary below is my own interpretation, not a result reported in the paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3tiC0hr_S2trdSbKutHkw.png" /><figcaption><em>An author’s interpretation of how the families trade off</em></figcaption></figure><ul><li><strong>Statistical time series:</strong> easy to explain, works on small data, gives uncertainty. Weak on nonlinearity and spatial structure.</li><li><strong>Geostatistical:</strong> easy to explain, strongly location-aware, works on small data. Weak on nonlinearity.</li><li><strong>Classical machine learning:</strong> strong on nonlinear relationships. Middling on explainability and small data, with no built-in uncertainty.</li><li><strong>Deep learning:</strong> strong on nonlinearity with some spatial awareness. Hard to explain, data-hungry, and without built-in uncertainty.</li><li><strong>Probabilistic and Bayesian:</strong> strong on uncertainty and small data, and reasonable across the rest.</li></ul><p>No single family wins every criterion, which is consistent with the paper’s own conclusion.</p><p><strong>Evidence from baseline comparisons</strong></p><p>As a survey, the paper contains no ablation studies of its own. What it does report is how the reviewed studies compared a method against a simpler baseline, and these are the closest available evidence:</p><ul><li><strong>Physics-guided LSTM vs plain LSTM:</strong> adding physical constraints raised R² by <strong>20.7%</strong> and reduced RMSE by <strong>8.2%</strong> in next-day forecasting.</li><li><strong>Graph-ConvLSTM vs ConvLSTM:</strong> adding a graph component reduced RMSE by up to <strong>14.3%</strong> on 7-day forecasts.</li><li><strong>Co-Kriging vs Kriging and IDW:</strong> adding SMOS soil moisture as a helper variable gave the best rainfall maps, with R² between 0.944 and 0.992.</li><li><strong>XGBoost with a genetic algorithm vs Random Forest, SVM, and CatBoost:</strong> best of the four, with R² of 0.891.</li><li><strong>Random Forest vs process-based models:</strong> the data-driven model was more accurate for root-zone moisture.</li><li><strong>ANN vs Ordinary Kriging for gap filling:</strong> the ANN performed better, especially in sparsely vegetated areas.</li><li><strong>Hierarchical vs regular Bayesian:</strong> more accurate drought forecasts with fewer false alarms.</li><li><strong>Gaussian process with a smarter sample design vs plain Gaussian process:</strong> lower error and shorter training time.</li><li><strong>Transformer vs LSTM:</strong> mixed results, with the LSTM ahead in one study and the transformer in another.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KgMz0c_7zlwVYHcpR0wv1A.png" /><figcaption><em>Reported gains from adding one ingredient, shown for scale only. These come from different studies and metrics and are not a leaderboard.</em></figcaption></figure><p>The physics-guided LSTM is the closest thing to a true ablation here, since it is the same model with one ingredient added. The remaining comparisons involve different models trained on different data, so they are better read as hints than as rankings.</p><p><strong>Evaluation metrics</strong></p><p>For regression, the survey lists R², RMSE, MAE, ubRMSE, and KGE. For classification, it lists accuracy, F1-score, kappa, precision, recall, and AUC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3YOgEZGJWmT8zLUktWgrA.png" /><figcaption><em>The standard metrics</em></figcaption></figure><p>No single metric captures every failure. A model can have a strong R² and still make costly errors at one depth or during a drought. For classification, accuracy alone can hide weak performance on a rare moisture class, so class-sensitive metrics are essential.</p><p><strong>Validation that reflects deployment</strong></p><p>The survey’s most practical recommendation concerns how data are split. Soil moisture is correlated in space and time, so random cross-validation can place near-identical neighbors on both sides of the split and overstate accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__e3J_ejU-0rYE3dTfdC7Q.png" /><figcaption><em>A random split leaks information between neighbors, while a blocked split tests on an unseen region</em></figcaption></figure><p>The split should resemble the intended use. Forecasting a future period calls for a temporal holdout, mapping a new region calls for spatial blocking, and trusting a new sensor calls for leave-one-station-out validation.</p><p><strong>Critical assessment</strong></p><p>The survey has clear strengths. Its taxonomy is clean, its pipeline treats data preparation and validation as first-class steps, and it is candid about why reported results cannot be ranked against one another: the studies differ in inputs, outputs, regions, depths, and validation designs.</p><p>I see four limitations:</p><ul><li><strong>No common benchmark.</strong> The summary table lists inputs and outputs for each method but offers no like-for-like performance comparison. The paper names the difficulty and stops there.</li><li><strong>Limited coverage of classification,</strong> even though drought and irrigation decisions often depend on categories.</li><li><strong>Interpretability and uncertainty are named as gaps but lightly explored.</strong> Physics-guided deep learning receives a single paragraph, and I would have liked far more, since it is the most promising trend discussed.</li><li><strong>Editorial polish.</strong> A few references are duplicated, there are minor typos, and one cited study predicts soil respiration rather than soil moisture.</li></ul><p>The most valuable next step for the field would be a shared benchmark dataset with fixed spatial and temporal splits, so that models can finally be compared on equal footing.</p><p><strong>Conclusion</strong></p><p>The survey does not name a single best soil moisture model, and its reasoning is sound. The most useful lesson is procedural: define the decision first, align the evidence carefully, respect space and time when testing, and report uncertainty when the output will drive action.</p><p>My practical takeaway is to begin with an explainable baseline such as Kriging or a Random Forest, and to add complexity only when a specific limitation justifies it. The fanciest model is not automatically the best one. The best model is the one tested under the conditions in which it will actually be used.</p><p><strong>References</strong></p><ul><li>Tsimpidi, I., Georgoulas, G., Sumathy, V., and Nikolakopoulos, G. (2026). “A Survey on Data-Driven Models for Soil Moisture Regression and Classification.” arXiv:2606.18316: <a href="https://arxiv.org/pdf/2606.18316">https://arxiv.org/pdf/2606.18316</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3fa2096adaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Accuracy Can Lie: Understanding Small-Sample Class Imbalance in Machine Learning]]></title>
            <link>https://medium.com/@9navya9/why-accuracy-can-lie-understanding-small-sample-class-imbalance-in-machine-learning-b0975e1bed89?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0975e1bed89</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 02:50:03 GMT</pubDate>
            <atom:updated>2026-10-07T03:50:47.714Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A data-centric reading of a 2025 survey on metrics, feature complexity, and resampling.</em></p><h3>A 99% model that misses every fraud case</h3><p>Imagine a fraud-detection dataset with 990 legitimate transactions and 10 fraudulent ones. A model that predicts “legitimate” for every transaction is 99% accurate. It also detects zero fraudulent transactions.</p><p>That is not a trick in the arithmetic. Accuracy is correctly reporting the fraction of all predictions that are right. The problem is that the fraction of correct predictions does not tell us which class the model is failing to recognize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2Jr7PowedD2nqsLTPzgdw.png" /><figcaption>Illustrative example of an accuracy trap</figcaption></figure><p>This is the central warning behind the small-sample imbalance problem. When data is limited and the classes are unevenly represented, a model can look successful while learning very little about the class that matters most.</p><p>In this article, I use Shuxian Zhao and colleagues’ 2025 survey, <em>A Survey on Small Sample Imbalance Problem: Metrics, Feature Analysis, and Solutions</em>, to explain how to diagnose the problem before choosing a metric, classifier, or resampling method.</p><h3>Small-sample imbalance is more than unequal class counts</h3><p>Small-sample imbalance combines two weaknesses:</p><ol><li>The dataset does not contain enough examples for stable generalization.</li><li>The examples are distributed unevenly across classes.</li></ol><p>The second issue creates a majority-class bias. The first makes every unusual, noisy, or mislabeled observation more influential. Together, they can produce an unstable decision boundary: a model may fit the available minority examples but fail when it sees a slightly different case.</p><p>The survey also describes related forms of imbalance:</p><ul><li><strong>Feature imbalance:</strong> important information is distributed unevenly across variables.</li><li><strong>Sample-quality imbalance:</strong> one class may contain more noise, outliers, or ambiguous labels.</li><li><strong>Temporal imbalance:</strong> class proportions or patterns change over time.</li><li><strong>Spatial imbalance:</strong> observations from different regions do not follow the same distribution.</li></ul><p>This is why matching the class counts is not automatically the same as solving the learning problem. Two datasets can have the same imbalance ratio but behave very differently if one has cleanly separated classes and the other has heavy overlap.</p><h3>What the 2025 survey contributes</h3><p>The survey is organized around a data-centric question: what characteristics of a small, imbalanced dataset should guide the modeling decision?</p><p>Table I reviews feature-analysis tools such as mean and variance, skewness and kurtosis, Shannon entropy, t-SNE, PCA, LDA, KL divergence, the Kolmogorov–Smirnov test, wavelet transforms, and Fourier transforms.</p><p>Table II groups data-complexity measures into four families:</p><ul><li><strong>Feature overlap:</strong> F1, F1v, F2, F3, F4, and IN</li><li><strong>Neighborhood structure:</strong> N1, N2, N3, N4, T1, and LSC</li><li><strong>Linear separability:</strong> L1, L2, and L3</li><li><strong>Other imbalance-complexity measures:</strong> CM, wCM, and dwCM</li></ul><p>These measures are useful because they ask a more informative question than “How unequal are the class counts?” They ask whether the classes overlap, whether local neighborhoods are mixed, whether a linear boundary is plausible, and whether the minority class is scattered into small disjuncts.</p><h3>The dataset names make the difficulty visible</h3><p>The survey uses examples from the UCI Machine Learning Repository and the KEEL Data Repository. Table III reports the following sample sizes, feature counts, class distributions, and imbalance ratios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOC4Ljq-5QRpN0FNIFoXQg.png" /></figure><p>The paper’s dataset identifiers are easy to overlook: the binary dataset is printed as ecol, while the multiclass dataset is printed as ecoli. I preserve those labels exactly because they identify rows in the survey’s tables; they are not a spelling change to the biological term <em>E. coli</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NN6TLzzJ8-9XjMeiaRfeA.png" /><figcaption>Imbalance ratios reported for the datasets in Table III</figcaption></figure><p>The range is substantial. The binary Wine dataset has an imbalance ratio of 1.51, while multiclass yeast reaches 92.60. A single “best” resampling recipe would have to work across radically different data conditions, which is unlikely without analyzing the data first.</p><h3>Choosing a metric means choosing what failure matters</h3><p>Table IV summarizes the metrics used in small-sample imbalanced classification. Each metric answers a different question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6OnsoEz7OyAQ_htsLtBbw.png" /></figure><p>The paper notes that precision-recall analysis can be more informative than the ROC curve when the minority class is very small. A ROC curve may look strong because the number of true negatives is large, even when the model’s positive predictions are poor.</p><p>The metric should therefore follow the application. A medical screening system may prioritize recall because missing a true case is costly. A system that sends alerts to human reviewers may prioritize precision to avoid overwhelming them with false alarms. In multiclass problems, macro-F1 prevents the largest class from dominating the summary score.</p><h3>Resampling is a hypothesis, not a button</h3><p>The survey reviews conventional methods such as SMOTE, Borderline-SMOTE, ADASYN, WK-SMOTE, MC-SMOTE, SUNDO, and WRO. It also covers synthetic-data generation with VAEs, GANs, and diffusion models; feature-selection methods; cost-sensitive learning; ensemble methods; and few-shot approaches for extreme imbalance.</p><p>Oversampling can give a classifier more minority examples to learn from, but synthetic points are not automatically realistic. Undersampling has the opposite risk: when the entire dataset is small, removing majority examples may discard useful structure. Both choices should be evaluated against a no-sampling baseline.</p><p>There is also a critical experimental rule: resampling must be applied only to the training portion of the data. If synthetic examples are created before the train-test split, information from the evaluation set can influence the training data and make the final score look better than it really is.</p><p>Table VII makes the trade-off visible. The reported values below are shown on the paper’s 0–100 scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cN6K8UsFKPa_LJaimTWKOw.png" /></figure><p>No method wins every column. FW-SMOTE has the highest reported G-Mean, 80.75, and the highest reported AUC, 86.40, while IA-SUWO has the highest reported F1-score, 71.67. The missing values for FW-SMOTE and RBO are not zeroes; the table does not report them.</p><h3>The experiments show why “the best method” is too simple</h3><p>The survey’s main comparison evaluates Decision Tree, SVM, Random Forest, k-Nearest Neighbors, and AdaBoost. It examines no sampling, SMOTE, Borderline-SMOTE, ADASYN, and additional complexity-aware strategies.</p><p>For the binary Breast_cancer dataset in Table X, the no-sampling accuracy values are:</p><ul><li>Decision Tree: <strong>0.9415</strong></li><li>SVM: <strong>0.9357</strong></li><li>Random Forest: <strong>0.9708</strong></li><li>KNN: <strong>0.9591</strong></li><li>AdaBoost: <strong>0.9766</strong></li></ul><p>The corresponding no-sampling AUC values are <strong>0.9438</strong>, <strong>0.9931</strong>, <strong>0.9968</strong>, <strong>0.9953</strong>, and <strong>0.9962</strong>, respectively.</p><p>For the multiclass Ecoli dataset in Table XI, the no-sampling accuracy values are:</p><ul><li>Decision Tree: <strong>0.7624</strong></li><li>SVM: <strong>0.8812</strong></li><li>Random Forest: <strong>0.8218</strong></li><li>KNN: <strong>0.8812</strong></li><li>AdaBoost: <strong>0.6535</strong></li></ul><p>The corresponding no-sampling AUC values are <strong>0.6967</strong>, <strong>0.9141</strong>, <strong>0.8386</strong>, <strong>0.8577</strong>, and <strong>0.6860</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-FzNmZ9IBn44UiM6nVFGg.png" /><figcaption>No-sampling accuracy reported in Tables X and XI</figcaption></figure><p>The difference between the two datasets is more revealing than any single leaderboard. SVM and KNN tie at 0.8812 accuracy on Ecoli, while AdaBoost is at 0.6535. On Breast_cancer, all five classifiers are above 0.93 accuracy before resampling.</p><p>The resampling results reinforce the same point:</p><ul><li>On Breast_cancer, the best AdaBoost F1-score shown is <strong>0.9825</strong> with SMOTE and SMOTE-IPF.</li><li>On Breast_cancer, the best G-Mean values shown include <strong>0.9828</strong> for AdaBoost with SMOTE and <strong>0.9782</strong> for KNN with SMOTE-ENN.</li><li>On Ecoli, Random Forest with ADASYN reaches the best F1-score shown, <strong>0.8753</strong>, and the best G-Mean shown, <strong>0.9266</strong>.</li><li>With SMOTE on Ecoli, the paper reports a <strong>29.7% accuracy difference</strong> between SVM and AdaBoost.</li><li>When the classifier is held constant on Breast_cancer, the Random Forest F1-score changes by only <strong>0.5 percentage points</strong> across the resampling methods.</li></ul><p>These results do not support a universal ranking of resampling techniques. They support a dataset–classifier interaction. A strong classifier on a relatively manageable dataset may gain little from a complicated sampler, while a more difficult multiclass dataset may respond differently to the same intervention.</p><h3>A practical workflow for small-sample imbalance</h3><p>The survey suggests a workflow that is more reliable than automatically applying SMOTE:</p><ol><li><strong>Quantify the imbalance.</strong> Record class counts, imbalance ratio, and — when relevant — multiclass measures such as adjusted IR or IBI³.</li><li><strong>Inspect the data structure.</strong> Examine feature distributions, overlap, outliers, missing values, neighborhood structure, and linear separability.</li><li><strong>Choose metrics from the cost of failure.</strong> Report class-level precision, recall, F1, G-Mean, AUC, or macro-averages instead of relying on accuracy alone.</li><li><strong>Build a no-sampling baseline.</strong> Compare several classifiers before introducing a data-processing intervention.</li><li><strong>Resample inside the training process.</strong> Put the sampler inside each training fold so that validation data remains untouched.</li><li><strong>Check stability.</strong> Repeat cross-validation or use multiple splits, then report variation instead of presenting one lucky score.</li><li><strong>Inspect the errors.</strong> Look at minority false negatives, false positives, class-specific results, and the examples created by synthetic sampling.</li><li><strong>Choose the simplest method that solves the observed problem.</strong> If the classifier already performs well, extra resampling may add complexity without meaningful benefit.</li></ol><p>This workflow turns resampling into a testable hypothesis: “the current failure is caused by insufficient minority coverage, and this method addresses it.” That statement can be evaluated. “SMOTE usually helps” cannot.</p><h3>What a deployed system would need beyond the paper’s tables</h3><p>The survey is focused on methods and evaluation, but its conclusions have a practical implication. A model should not be monitored only through a single aggregate score. A production system should track:</p><ul><li>class-specific precision and recall;</li><li>the frequency of minority false negatives;</li><li>confidence calibration and low-confidence predictions;</li><li>changes in class proportions and feature distributions;</li><li>performance on new or unfamiliar examples; and</li><li>the model version, sampling method, and training data used for each release.</li></ul><p>If the minority class is rare because the organization is not collecting enough representative data, resampling alone cannot solve the underlying evidence problem. Improving labels, collecting targeted examples, and reviewing ambiguous cases may be more valuable than generating more synthetic points.</p><h3>Conclusion: accuracy is not wrong — it is incomplete</h3><p>Accuracy is not lying. It is answering a narrower question than many machine-learning projects actually need.</p><p>The small-sample imbalance problem shows why reliable evaluation must connect four things: the data distribution, the model, the metric, and the cost of being wrong. The survey’s Table III datasets range from modest imbalance in Wine to extreme imbalance in multiclass yeast. Its Tables X and XI show that classifier performance changes across datasets, and Table VII shows that resampling methods trade off G-Mean, F1-score, and AUC rather than producing one universal winner.</p><p>The practical lesson is simple: diagnose first, establish a baseline, evaluate the minority class directly, and resample only when the data supports that decision. A trustworthy model is not the one with the most impressive overall percentage. It is the one whose important failures are visible, measured, and addressed.</p><h3>References</h3><ul><li>Zhao, S., Gui, J., Dong, M., Yu, B., Gui, Z., Dong, L., Tang, Y. Y., and Kwok, J. T.-Y. <a href="https://arxiv.org/pdf/2504.14800">“A Survey on Small Sample Imbalance Problem: Metrics, Feature Analysis, and Solutions.”</a> arXiv:2504.14800, 2025.</li><li><a href="https://archive.ics.uci.edu/">UCI Machine Learning Repository</a></li><li><a href="https://sci2s.ugr.es/keel/datasets.php">KEEL Data-Mining Software Tool</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0975e1bed89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching a Neural Network to Recognize Natural Scenes: From a Custom CNN to Explainable Transfer…]]></title>
            <link>https://medium.com/@9navya9/teaching-a-neural-network-to-recognize-natural-scenes-from-a-custom-cnn-to-explainable-transfer-b404798e954a?source=rss-fa313d1ceba7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b404798e954a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Navya Illa]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 06:10:48 GMT</pubDate>
            <atom:updated>2026-09-06T00:00:19.749Z</atom:updated>
            <content:encoded><![CDATA[<h3>Teaching a Neural Network to Recognize Natural Scenes: From a Custom CNN to Explainable Transfer Learning</h3><p>When a person sees a photograph of a forest or a city street, recognition feels immediate. A computer receives only a grid of pixel values. It must learn which visual patterns separate trees from buildings, glaciers from mountains, and coastlines from other landscapes.</p><p>In this project, I used chatbot-assisted coding to build an end-to-end deep-learning workflow for natural-scene classification. I compared a custom convolutional neural network with an ImageNet-pretrained MobileNetV2 model, fine-tuned the stronger approach, analyzed its errors, and used Grad-CAM to inspect the image regions influencing difficult predictions.</p><p>The final model achieved <strong>90.27% test accuracy</strong> and <strong>90.48% macro-F1</strong> across six classes. More importantly, its failures revealed where visual categories overlap and why aggregate accuracy is only the beginning of model evaluation.</p><p><strong>The dataset and the question</strong></p><p>I used the Intel Image Classification dataset from Kaggle. It contains six categories: buildings, forest, glacier, mountain, sea, and street.</p><p>The working data was divided into 11,228 training images, 2,806 validation images, and 3,000 untouched test images. The classes were reasonably balanced, so accuracy was useful, but I also calculated precision, recall, and macro-F1 to give each category equal importance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SuqZJhAoxLP-Ky5JfvLypQ.png" /><figcaption>Class distribution across the training and test sets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z5kjb2_KCtpo1a2zHLbhzA.png" /><figcaption>Representative images from the six scene categories</figcaption></figure><p>The objective was to answer three questions:</p><p>1. How well can a CNN trained from scratch recognize these scenes?<br>2. How much does transfer learning improve generalization?<br>3. What can the model’s errors and Grad-CAM maps teach us about its behavior?</p><p><strong>Following CRISP-DM for computer vision</strong></p><p>I organized the project using CRISP-DM: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment.</p><p>The business framing was intentionally broader than maximizing a leaderboard score. Scene recognition can support photo organization, visual search, and content indexing, but a deployed system also needs reliable confidence estimates, interpretable failure analysis, and monitoring for unfamiliar inputs.</p><p>During data understanding, I audited the folder structure, counted images by class and split, verified label names, and checked that the test set remained separate from training decisions.</p><p>For preparation, every image was resized to 150 × 150 pixels. Training-only augmentation introduced horizontal flips, small rotations, zoom, and contrast variation. These transformations encouraged the model to learn scene structure instead of memorizing exact training images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4aqXzp3-WdqmV891t4M-A.png" /><figcaption>Training-only augmentation examples</figcaption></figure><p>The test dataset was loaded without shuffling. This small implementation detail preserved the alignment between file paths, true labels, and predictions during error analysis.</p><p><strong>Baseline: a custom convolutional neural network</strong></p><p>The baseline was a CNN built from scratch with four convolutional blocks. Each block combined convolution, batch normalization, and max pooling. Global average pooling and dropout reduced overfitting before the six-class softmax output.</p><p>The custom CNN achieved <strong>80.10% test accuracy</strong>. Its learning curves showed genuine learning, but validation performance fluctuated and remained substantially below the training curve. That gap suggested limited generalization and motivated transfer learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6-OsCQ1IrTTTUkRKNwbNdg.png" /><figcaption>Custom CNN training and validation performance</figcaption></figure><p><strong>Transfer learning with MobileNetV2</strong></p><p>Instead of asking a new model to learn every visual feature from scratch, transfer learning begins with representations learned from a much larger dataset. I loaded MobileNetV2 with ImageNet weights, removed its original classification head, and attached a new global-average-pooling, dropout, and six-class softmax head.</p><p>The first stage froze the pretrained feature extractor and trained only 7,686 new parameters. I then unfroze the final 30 MobileNetV2 layers, kept batch-normalization layers frozen for stability, and fine-tuned approximately 1.5 million parameters using a learning rate of 0.00001.</p><p>Training ran on two NVIDIA T4 GPUs using TensorFlow’s `MirroredStrategy`. Early stopping restored the most useful weights, while learning-rate reduction allowed smaller adjustments when validation loss stopped improving.</p><p>The saved, fully rerun notebook produced the following test results:</p><p><strong>Custom CNN:</strong> 80.10% test accuracy with 0.5698 test loss.<br><strong>Fine-tuned MobileNetV2:</strong> 90.27% test accuracy with 0.2487 test loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mr6NHnicBgY8avXzpOcAXw.png" /><figcaption>Transfer learning improved test accuracy from 80.10% to 90.27%</figcaption></figure><p>Transfer learning improved test accuracy by <strong>10.17 percentage points</strong>. This is not simply the result of a larger model. It reflects the value of reusable edge, texture, shape, and object representations learned from diverse images.</p><p><strong>Looking beyond accuracy</strong></p><p>The final model’s macro precision was 90.43%, macro recall was 90.63%, and macro-F1 was 90.48%. Forest was the easiest class, reaching 98.95% F1. Glacier and mountain were the hardest, with F1 scores of 83.15% and 84.24%.</p><p>The confusion matrix explained why. The largest directional errors were:</p><p>- 69 mountain images classified as glacier<br>- 63 glacier images classified as mountain<br>- 49 street images classified as buildings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_RNAdNxUn9DEhzKiSYL06Q.png" /><figcaption>Count and normalized confusion matrices</figcaption></figure><p>These mistakes are visually plausible. Mountain and glacier photographs often share snow, exposed rock, distant peaks, and similar horizons. Streets and buildings frequently occur in the same urban image, so the correct label may depend on which part of the scene dominates.</p><p><strong>The value of confident mistakes</strong></p><p>I examined the model’s most confident incorrect predictions. Several exceeded 99% confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BrI-Xyg3JrkqcGMIpYN3-Q.png" /><figcaption>Highly confident incorrect predictions</figcaption></figure><p>This is an important warning: softmax confidence is not the same as correctness. A classifier can be highly certain when an image contains features strongly associated with the wrong category. A deployed system should therefore monitor calibration and consider deferring low-confidence or unusual cases to a human reviewer.</p><p>The error gallery also raises a dataset question. Some images legitimately contain multiple concepts. A photograph dominated by buildings may be labeled street, while a snowy mountain scene could reasonably resemble a glacier. Error analysis should examine label ambiguity rather than assuming every disagreement is purely a model failure.</p><p><strong>Explaining predictions with Grad-CAM</strong></p><p>Grad-CAM produces a heatmap showing spatial regions that most influenced a convolutional model’s selected prediction. I applied it to six highly confident errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IheQKMozx3a5cuuIXJtYIQ.png" /><figcaption>Grad-CAM regions influencing confident model errors</figcaption></figure><p>The model often emphasized recognizable content: towers and facades for buildings, horizons and water for sea, and snowy rock formations for glacier. These maps made the mistakes easier to understand. For example, when a mountain was predicted as glacier, the strongest activation often covered its snow-covered region.</p><p>Grad-CAM must be interpreted cautiously. It shows sensitivity within the model, not human-like reasoning, causal proof, or a guarantee that the highlighted area is semantically correct.</p><p><strong>What an actual deployment would require</strong></p><p>A lightweight application could accept an uploaded photograph, resize and preprocess it, and return the six class probabilities. The trained `.keras` model is already preserved as an artifact.</p><p>Production readiness would require more than serving predictions. I would monitor:</p><p>- Input drift and changing image characteristics<br>- Confidence calibration<br>- Class-specific precision and recall<br>- Frequency of low-confidence predictions<br>- Performance on images outside the six known classes<br>- Latency, memory use, and model-version provenance</p><p>The system should expose uncertainty and avoid forcing every unfamiliar image into a known category. An out-of-distribution or rejection mechanism would be a valuable next improvement.</p><p><strong>How AI assistance changed the workflow</strong></p><p>ChatGPT/Codex assisted with planning the CRISP-DM stages, generating TensorFlow code, diagnosing disabled GPU and internet settings, building the custom CNN and MobileNetV2 workflow, interpreting metrics, and structuring the final report.</p><p>AI assistance accelerated iteration, but verification remained essential. Every code block was executed in Kaggle, errors were corrected using observed traces, and final claims were reconciled against exported CSV and JSON results. The complete notebook, prompts, model files, figures, and metrics are preserved in the GitHub repository.</p><p>One subtle lesson came from reproducibility. The interactive training session reached 90.83% test accuracy, while the clean saved rerun reached 90.27%. Small differences can remain even with a fixed random seed when training across multiple GPUs. I therefore used the clean rerun’s 90.27% result everywhere in the final report.</p><p><strong>Conclusion</strong></p><p>The project demonstrated that transfer learning can make a decisive difference on a moderate-sized image dataset. MobileNetV2 improved test accuracy by more than ten percentage points over the custom CNN while supporting detailed class-level and spatial error analysis.</p><p>The most valuable outcome was not a single accuracy number. It was a reproducible workflow connecting data preparation, model design, evaluation, explainability, and deployment considerations. A useful deep-learning project should show not only what the model gets right, but also where it fails, why those failures are plausible, and how its behavior would be monitored beyond the notebook.</p><p><strong>References</strong></p><p>- Intel Image Classification dataset: <a href="https://www.kaggle.com/datasets/puneet6060/intel-image-classification">https://www.kaggle.com/datasets/puneet6060/intel-image-classification</a><br>- TensorFlow image classification tutorial: <a href="https://www.tensorflow.org/tutorials/images/classification">https://www.tensorflow.org/tutorials/images/classification</a><br>- Keras MobileNetV2 documentation: <a href="https://keras.io/api/applications/mobilenet/">https://keras.io/api/applications/mobilenet/</a><br>- Sandler et al. (2018), “MobileNetV2: Inverted Residuals and Linear Bottlenecks”: <a href="https://arxiv.org/abs/1801.04381">https://arxiv.org/abs/1801.04381</a><br>- Selvaraju et al. (2017), “Grad-CAM”: <a href="https://arxiv.org/abs/1610.02391">https://arxiv.org/abs/1610.02391</a><br>- Wirth and Hipp (2000), “CRISP-DM”: <a href="https://www.the-modeling-agency.com/crisp-dm.pdf">https://www.the-modeling-agency.com/crisp-dm.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b404798e954a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>