<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashish Subedi on Medium]]></title>
        <description><![CDATA[Stories by Ashish Subedi on Medium]]></description>
        <link>https://medium.com/@ashish-subedi?source=rss-f782bff94d60------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ykFUXXkqkzNjrr26</url>
            <title>Stories by Ashish Subedi on Medium</title>
            <link>https://medium.com/@ashish-subedi?source=rss-f782bff94d60------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:31:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashish-subedi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why an Automatic Diesel Sells for 3.5×]]></title>
            <link>https://ashish-subedi.medium.com/why-an-automatic-diesel-sells-for-3-5-ef2f11668be9?source=rss-f782bff94d60------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef2f11668be9</guid>
            <dc:creator><![CDATA[Ashish Subedi]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 11:48:06 GMT</pubDate>
            <atom:updated>2026-06-04T12:15:54.552Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Why an Automatic Diesel Sells for 3.5× More Than a Manual Petrol — and How Linear Regression Proved It</strong></h3><p><em>A complete machine learning walkthrough: data cleaning, feature engineering, log transformation, and what the numbers actually reveal about India’s used car market.</em></p><h4>The Question</h4><p>I have watched people get completely different prices for the same car model. Same city., Same mileage, But the selling price? Nowhere close. And neither the buyer nor the seller could clearly explain why.</p><p>Can Linear Regression accurately predict used car selling prices based on fuel type, transmission, age, mileage, and brand?</p><p>Everyone assumes newer cars sell for more. Everyone assumes Diesel beats Petrol. Everyone assumes low mileage means high price. But by exactly <em>how much</em>? And how do these factors combine to produce the final rupee number?</p><h4>The Dataset</h4><p>4,340 used car listings. 7 features. One target variable.</p><pre>| Feature       | Type        | Description                                   |<br>| ------------- | ----------- | --------------------------------------------- |<br>| name          | Categorical | Full car name (brand extracted from this)     |<br>| year          | Numeric     | Manufacturing year                            |<br>| selling_price | Numeric     | Listed selling price in INR (target variable) |<br>| km_driven     | Numeric     | Total kilometres driven                       |<br>| fuel          | Categorical | Petrol, Diesel, CNG, LPG, Electric            |<br>| seller_type   | Categorical | Individual, Dealer, Trustmark Dealer          |<br>| transmission  | Categorical | Manual or Automatic                           |<br>| owner         | Categorical | First Owner through Fourth &amp; Above Owner      |</pre><p>Selling prices range from ₹20,000 to ₹89,00,000 with a mean of ₹5,04,127 and a standard deviation of ₹5,78,549.</p><p>When your standard deviation is larger than your mean, the distribution is not normal. That is the first signal a log transform will be needed.</p><h4>Step 1: Data Loading and Exploration</h4><p>First look at the data revealed something important — it is clean.</p><pre>Shape         : (4340, 8)<br>Missing Values: 0 across all columns</pre><p>No nulls. No corrupted entries. But the describe output told an important story. Selling price had a max of ₹89,00,000 against a median of just ₹3,50,000 — a 25× gap. The dataset spans everything from a weathered 2007 Maruti 800 to a pristine Land Rover. Cars range from 1992 to 2020, nearly three decades of listings in one file.</p><h4>Step 2: Data Cleaning</h4><p>Two things to check before touching anything — outliers and duplicates.</p><pre>selling_price : 271 outliers  (keeping — natural skew in used-car market)<br>km_driven     : 110 outliers  (keeping — natural skew in used-car market)<br>Duplicate rows removed : 763<br>Dataset shape after cleaning : (3577, 8)</pre><p>763 duplicate rows removed — that is 17.6% of the raw dataset. These are identical listings, almost certainly the same car listed multiple times by the same seller. Keeping them would bias the model towards the most-frequently listed cars and silently inflate every metric.</p><p>The outliers are a different matter. A 2012 Land Rover with 40,000 km is a genuine premium listing. Removing it because the price is extreme would teach the model nothing about high-value cars — the exact segment where accurate pricing matters most. Both outlier groups were kept. Only duplicates were dropped.</p><h4>Step 3: Data Visualization (EDA)</h4><h4>Selling Price Distribution and Fuel Type vs Selling Price</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7FdhnFQQ9PSsq_bz.png" /></figure><p>The selling price histogram confirms what the statistics already suggested — extreme right skew. Most cars sell between ₹1 lakh and ₹8 lakhs. A long tail of premium listings stretches all the way to ₹89 lakhs.</p><p>The fuel type boxplot tells the first major story. Diesel cars fetch nearly double the median price of Petrol cars. The median Diesel listing sits above the 75th percentile of Petrol listings. Diesel’s combination of fuel efficiency and highway torque commands a clear premium in the Indian used car market.</p><pre>| Fuel Type | Average Selling Price |<br>| --------- | --------------------- |<br>| Diesel    | ₹6,69,094             |<br>| Petrol    | ₹3,44,840             |<br>| CNG       | ₹2,77,175             |<br>| LPG       | ₹1,67,826             |</pre><h4>KM Driven vs Selling Price and Correlation Matrix</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Qq6YuGVNfcHwJSc.png" /></figure><p>The scatter plot of km driven versus selling price reveals the expected negative trend — more mileage, lower price — but with enormous spread. A 60,000 km car could be worth ₹2 lakh or ₹25 lakh depending on every other variable.</p><p>The correlation matrix quantifies it:</p><pre>| Feature   | Correlation with Selling Price |<br>| --------- | ------------------------------ |<br>| year      | **+0.41**                      |<br>| km_driven | −0.19                          |</pre><p>Year is the strongest numeric predictor by a wide margin. Newer cars cost more — that confirms intuition. km_driven has a negative relationship but weaker than expected. A low-mileage 2008 car is still worth less than a high-mileage 2018 car. Age dominates mileage.</p><p>The transmission boxplot (not shown separately) told the sharpest story:</p><pre>| Transmission | Average Selling Price |<br>| ------------ | --------------------- |<br>| Automatic    | **₹14,08,154**        |<br>| Manual       | ₹4,00,067             |</pre><p>Automatics average 3.5× more than Manuals. Automatic transmission is still a premium feature in India, correlating heavily with higher-segment brands and urban buyers who pay for convenience.</p><h4>Step 4: Feature Engineering</h4><p>Raw features alone are not enough. Three new features were created to give the model richer signal.</p><p>car_age — derived from year:</p><pre>car_age = 2024 - year</pre><p>This converts an absolute year (2015) into a relative measure of depreciation (9 years old). Linear models handle relative age far better than calendar years, which carry no inherent meaning to a coefficient.</p><p>brand — extracted from the car name:</p><pre>&quot;Hyundai Creta 1.6 CRDi SX Option&quot;  →  &quot;Hyundai&quot;</pre><p>The full name is noise for a linear model. Brand captures the premium vs budget signal cleanly. Brands with fewer than 50 listings were grouped into &quot;Other&quot; to avoid rare categories creating unstable one-hot columns.</p><p>km_driven_log — log transform of mileage:</p><pre>km_driven_log = log1p(km_driven)</pre><p>Kilometers driven has its own right skew. The difference between 10,000 and 20,000 km matters. The difference between 4,00,000 and 4,10,000 km does not. Log scaling compresses the tail and makes the mileage relationship more linear.</p><p>Brands with 50+ listings included in the model:</p><pre>| Brand      | Listings |<br>| ---------- | -------- |<br>| Maruti     | 1,280    |<br>| Hyundai    | 821      |<br>| Mahindra   | 365      |<br>| Tata       | 361      |<br>| Honda      | 252      |<br>| Ford       | 238      |<br>| Toyota     | 206      |<br>| Renault    | 146      |<br>| Volkswagen | 107      |<br>| Skoda      | 68       |<br>| Nissan     | 64       |</pre><p>Everything else — Audi, BMW, Mercedes, Land Rover — went into &quot;Other&quot;. Too few listings to learn reliable price coefficients from.</p><h4>Step 5: Preprocessing and Encoding</h4><p>Machine learning models do not read strings. Every categorical variable became a number.</p><p>Binary encoding for two-category columns:</p><pre>transmission : Manual → 0,     Automatic → 1<br>seller_type  : Individual → 0,  Dealer → 1</pre><p>Ordinal encoding for ownership — this one carries a natural order that must be preserved:</p><pre>Test Drive Car        → 0<br>First Owner           → 1<br>Second Owner          → 2<br>Third Owner           → 3<br>Fourth &amp; Above Owner  → 4</pre><pre>| Owner Type           | Average Price |<br>| -------------------- | ------------- |<br>| Test Drive Car       | ₹9,54,294     |<br>| First Owner          | ₹5,98,637     |<br>| Second Owner         | ₹3,43,891     |<br>| Third Owner          | ₹2,69,474     |<br>| Fourth &amp; Above Owner | ₹1,73,901     |</pre><p>The price drops steadily with each ownership change — exactly what ordinal encoding captures. One-hot encoding would throw that ranking away and treat the categories as arbitrary and unrelated.</p><p>One-hot encoding for fuel and brand (drop_first=True to avoid multicollinearity):</p><pre>fuel  → fuel_Diesel, fuel_Electric, fuel_LPG, fuel_Petrol<br>brand → brand_Ford, brand_Honda, brand_Hyundai ... brand_Volkswagen</pre><p>Final shape after encoding: (3577, 21) — 20 features ready for modelling.</p><h4>Step 6: Log Transform</h4><pre>Selling price skew BEFORE log :  5.4539<br>Selling price skew AFTER log  : -0.0230</pre><h4>Selling Price Before and After Log Transform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aksSt0X4lBxEB6FB.png" /></figure><p>A skewness of 5.45 is extreme. A normal distribution has skewness zero. Anything above 1.0 is considered highly skewed. At 5.45, the distribution is so right-tailed that linear regression’s core assumption — that errors are normally distributed around predictions — is completely violated.</p><p>Training on raw prices would let a handful of ₹50–89 lakh premium listings dominate the cost function while the model learns almost nothing about the ₹2–8 lakh cars that make up 75% of the market.</p><p>Log transform collapses the tail. Skewness drops from 5.45 to −0.02 — essentially zero. The distribution becomes symmetric and the regression can now learn stable relationships across the full price range without being pulled toward extreme values.</p><p>Before: a right-skewed histogram with a long tail. After: a clean bell curve centered around the mean. One line of code. The single most impactful preprocessing step in the entire pipeline.</p><h4>Step 7: Train/Test Split and Feature Scaling</h4><pre>Train size : 2,861 rows<br>Test size  :   716 rows<br>Features   :    20</pre><p>80/20 split with random_state = 42 for reproducibility.</p><p>StandardScaler was fitted on training data only, then used to transform both sets. This distinction matters more than it sounds.</p><p>Fitting the scaler on the full dataset before splitting leaks test set statistics into training. The model learns the scale of data it was never supposed to see. Results look cleaner than they are. That mistake only reveals itself in production when the model meets genuinely unseen listings.</p><h4>Step 8: Base Model and Linear Regression</h4><p>Every ML project needs a baseline — the dumbest possible benchmark the model must beat to prove it learned anything at all.</p><p>Base Model: always predict the mean log-price</p><pre>Base Model R²   = -0.0000<br>Base Model RMSE =  0.8281</pre><p>R² of zero means it explains none of the variance. This is the floor. Anything the Linear Regression model scores above this is genuine learning.</p><p>Linear Regression Results (log scale)</p><pre>Train RMSE =  0.4114<br>Test RMSE  =  0.4124<br>Test MAE   =  0.3235<br>R²         =  0.7519<br>Adj R²     =  0.7448</pre><p>R² of 0.7519 means the model explains 75% of used car price variation from 20 features derived from 7 original columns. Against a base model that explains 0%, that is three-quarters of the entire prediction problem solved by a single linear equation.</p><p>Adjusted R² of 0.7448 confirms performance holds even after penalizing for 20 features. Every feature is earning its place — none of them are inflating the score with noise.</p><h4>Step 9: Overfitting Check and Cross Validation</h4><p>A model that performs well on training data and fails on new listings is not a model. This step exists to catch that failure.</p><p>Overfitting and Underfitting Analysis</p><pre>Train RMSE =  0.4114<br>Test RMSE  =  0.4124<br>Gap        = +0.2%<br>Diagnosis  = GOOD FIT</pre><p>A gap of +0.2%. Not 20%. Not 5%. 0.2%. Train error and test error are essentially identical. The model learned the underlying pricing patterns rather than memorizing the training listings. This is the healthiest possible result.</p><p>5-Fold Cross Validation</p><pre>CV RMSE = 0.4143 ± 0.0070</pre><p>A standard deviation of 0.0070 across 5 folds is exceptionally tight. Regardless of which 20% of the data the model is evaluated on, performance barely moves. Low variance in cross validation is the most reliable signal that results will hold on cars the model has never seen.</p><h4>Step 10: Final Results on Original Rupee Scale</h4><p>All metrics above were on log-transformed prices. Necessary for training. Not useful for explaining to a seller how far the prediction was off in rupees.</p><p>Converting predictions back from log scale gives the real picture.</p><h4>Actual vs Predicted Prices and Residuals</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6QR6uWEw10P8liMj.png" /></figure><p>The actual vs predicted scatter shows most listings clustered close to the red diagonal — the line of perfect prediction. Deviations concentrate at the high end. The model underestimates prices for premium cars, which is expected: there are only a handful of ₹50+ lakh listings in the dataset, far too few to learn their pricing patterns with full confidence.</p><p>The residuals plot shows errors scattered randomly around zero. No systematic pattern. Random residuals confirm the model captured the structural relationships in the data. What remains is genuine noise — the unpredictable part that depends on negotiation, physical condition, and market timing.</p><p>Final Metrics (Original Rupee Scale)</p><pre>| Metric                  | Value           |<br>| ----------------------- | --------------- |<br>| Training Samples        | 2,861           |<br>| Test Samples            | 716             |<br>| Features Used           | 20              |<br>| R² (log scale)          | 0.7519          |<br>| Adjusted R² (log scale) | 0.7448          |<br>| R² (original scale)     | 0.5333          |<br>| RMSE (original scale)   | ₹3,87,736       |<br>| MAE (original scale)    | ₹1,61,753       |<br>| Fit Diagnosis           | GOOD FIT        |<br>| Cross-Validation RMSE   | 0.4143 ± 0.0070 |</pre><p>R² drops from 0.75 to 0.53 when converting from log to original scale. This is expected and honest. Log scale smooths the extreme values. Original scale exposes them. Both numbers are correct — 0.75 measures model quality on normalized data, 0.53 measures real-world accuracy in rupees.</p><p>MAE of ₹1,61,753 means the typical prediction is off by about ₹1.6 lakh. For a market where prices range from ₹20,000 to ₹89,00,000 — a 445× spread — that is a reasonable margin for a single linear equation trained on 7 original features.</p><h3>Final Thoughts</h3><p>Used car prices in India are not random. They follow patterns driven by manufacturing year, brand, fuel type, transmission, and ownership history. Linear Regression — with the right cleaning, feature engineering, ordinal encoding, and log transformation — captures 75% of that structure from just seven original input variables.</p><p>The remaining 25% lives in data that was never collected. Physical condition. Service history. Accident records. Regional demand fluctuations. Inspection reports. These factors matter. They simply are not in this dataset.</p><p>What was in this dataset was used fully. For a baseline linear model operating on a market with a 445× price spread from bottom to top, an R² of 0.75 on log scale and a cross-validation standard deviation of 0.0070 is a result worth building on.</p><p>Data Source:<a href="https://www.kaggle.com/datasets/nehalbirla/vehicle-dataset-from-cardekho"> CarDekho Used Car Listings Dataset (Kaggle)</a></p><p>GitHub: [<a href="https://github.com/sokebat/car-price-prediction-linear-regression">car-price-prediction-linear-regression</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef2f11668be9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text Cleaning : How I Sanitized Real Reddit Posts (and Why You Should Too)]]></title>
            <link>https://ashish-subedi.medium.com/text-cleaning-for-nlp-how-i-sanitized-real-reddit-posts-and-why-you-should-too-45134009be46?source=rss-f782bff94d60------2</link>
            <guid isPermaLink="false">https://medium.com/p/45134009be46</guid>
            <dc:creator><![CDATA[Ashish Subedi]]></dc:creator>
            <pubDate>Mon, 11 May 2026 17:49:31 GMT</pubDate>
            <atom:updated>2026-05-11T17:53:18.232Z</atom:updated>
            <content:encoded><![CDATA[<p>Raw text from the wild is chaos. Before you feed it to any model — sentiment analysis, topic modeling, or even an LLM — you must clean it. This post walks through a complete, production‑ready text‑cleaning pipeline applied to real posts scraped from <strong>r/datascience</strong>. You’ll see exactly what noise looks like, how to diagnose it, and how to build a modular cleaning pipeline that preserves meaning while removing garbage.</p><h3>1. The Problem: Raw Text Is a Mess</h3><p>I recently scraped 86 posts from Reddit’s r/datascience using a custom scraper. The data came as a JSON array with fields like id, title, author, score, and — most importantly — selftext (the body of the post).</p><pre> {<br>    &quot;source&quot;: &quot;pullpush_submission_search&quot;,<br>    &quot;subreddit&quot;: &quot;datascience&quot;,<br>    &quot;id&quot;: &quot;1ki9zo3&quot;,<br>    &quot;title&quot;: &quot;Client told me MS Copilot replicated what I built. It didn’t.&quot;,<br>    &quot;author&quot;: &quot;melissa_ingle&quot;,<br>    &quot;selftext&quot;: &quot;I built three MVP models for a client over 12 weeks. Nothing fancy: an LSTM, a prophet model, and XGBoost. The difficulty, as usual, was getting and understanding the data and cleaning it. The company is largely data illiterate. Turned in all 3 models, they loved it then all of a sudden canceled the pending contract to move them to production. Why? They had a devops person do in MS Copilot Analyst (a new specialized version of MS Copilot studio) and it took them 1 week! Would I like to sign a lesser contract to advise this person though? I finally looked at their code and it’s 40 lines of code using a subset of the California housing dataset run using a Random Forest regressor. They had literally nothing. My advice to them: go f*%k yourself.&quot;,<br>    &quot;score&quot;: 1,<br>    &quot;upvote_ratio&quot;: 1.0,<br>    &quot;num_comments&quot;: 0,<br>    &quot;created_utc&quot;: &quot;2025-05-09T04:28:25+00:00&quot;,<br>    &quot;permalink&quot;: &quot;https://reddit.com/r/datascience/comments/1ki9zo3/client_told_me_ms_copilot_replicated_what_i_built/&quot;,<br>    &quot;link_flair_text&quot;: &quot;ML&quot;,<br>    &quot;is_self&quot;: true,<br>    &quot;domain&quot;: &quot;self.datascience&quot;<br>  }</pre><p>That looks okay, right? But many other posts contained:</p><ul><li>Markdown (**bold**, [link](url), &gt; quotes)</li><li>HTML entities (&amp;amp;, &amp;lt;)</li><li>URLs (https://t.co/...)</li><li>Repeated letters (sooooo, yessss!!!)</li><li>Unicode oddities (ﬁ instead of fi, curly quotes)</li><li>Internet slang (idk, tbh, yoe)</li></ul><p>Text cleaning is the process of transforming this raw, human‑messy input into a consistent, structured form that machine learning models can actually learn from.</p><h3>2. Diagnose Before You Clean</h3><p>Before writing a single regex, you must measure what kind of noise actually exists in your corpus. I wrote a small inspect_text() function that flags common issues:</p><pre>import re<br><br>def inspect_text(text: str) -&gt; dict:<br> return {<br>        &#39;length&#39;: len(text),<br>        &#39;word_count&#39;: len(text.split()),<br>        &#39;has_uppercase&#39;: bool(re.search(r&#39;[A-Z]&#39;, text)),<br>        &#39;has_extra_spaces&#39;: &#39;  &#39; in text,<br>        &#39;has_new_lines&#39;: &#39;\n&#39; in text,<br>        &#39;has_tabs&#39;: &#39;\t&#39; in text,<br>        &#39;has_urls&#39;: bool(re.search(r&#39;https?://\S+|www\.\S+&#39;, text)),<br>        &#39;has_emails&#39;: bool(re.search(r&#39;\b[\w\.-]+@[\w\.-]+\.\w+\b&#39;, text)),<br>        &#39;has_hashtags&#39;: bool(re.search(r&#39;#\w+&#39;, text)),<br>        &#39;has_mentions&#39;: bool(re.search(r&#39;@\w+&#39;, text)),<br>        &#39;has_html_tags&#39;: bool(re.search(r&#39;&lt;[^&gt;]+&gt;&#39;, text)),<br>        &#39;has_markdown&#39;: bool(re.search(r&#39;\*\*|__|\[.*?\]\(.*?\)&#39;, text)),<br>        &#39;has_repeated_letters&#39;: bool(re.search(r&#39;(.)\1{2,}&#39;, text)),<br>        &#39;has_digits&#39;: bool(re.search(r&#39;\d&#39;, text)),<br>        &#39;has_non_ascii&#39;: any(ord(ch) &gt; 127 for ch in text),<br>        &#39;punctuation_count&#39;: len(re.findall(r&#39;[^\w\s]&#39;, text)),<br>    }<br><br>#output look like this<br><br>{&#39;has_digits&#39;: True,<br> &#39;has_emails&#39;: False,<br> &#39;has_extra_spaces&#39;: False,<br> &#39;has_hashtags&#39;: False,<br> &#39;has_html_tags&#39;: False,<br> &#39;has_markdown&#39;: False,<br> &#39;has_mentions&#39;: False,<br> &#39;has_new_lines&#39;: False,<br> &#39;has_non_ascii&#39;: True,<br> &#39;has_repeated_letters&#39;: False,<br> &#39;has_tabs&#39;: False,<br> &#39;has_uppercase&#39;: True,<br> &#39;has_urls&#39;: False,<br> &#39;length&#39;: 751,<br> &#39;punctuation_count&#39;: 23,<br> &#39;word_count&#39;: 136}</pre><h3>3. The Cleaning Toolkit (One Function per Job)</h3><p>I built each cleaning operation as a small, testable function. This makes the pipeline transparent and easy to modify.</p><h4>3.1 Lowercasing</h4><p>The single highest‑impact step. Collapses case variants so Python, python, and PYTHON become the same token.</p><pre>def to_lowercase(text: str) -&gt; str:<br>    return text.lower()</pre><h4>3.2 Removing Markdown</h4><p>Reddit uses Markdown for formatting. The syntax adds zero semantic value for most NLP tasks.</p><pre>def remove_markdown(text: str) -&gt; str:<br>    # bold / italic<br>    text = re.sub(r&#39;\*{1,2}(.*?)\*{1,2}&#39;, r&#39;\1&#39;, text)<br>    text = re.sub(r&#39;_{1,2}(.*?)_{1,2}&#39;, r&#39;\1&#39;, text)<br>    # links: [text](url) → text<br>    text = re.sub(r&#39;\[([^\]]+)\]\([^)]+\)&#39;, r&#39;\1&#39;, text)<br>    # blockquotes and headings<br>    text = re.sub(r&#39;^&gt;+.*$&#39;, &#39; &#39;, text, flags=re.MULTILINE)<br>    text = re.sub(r&#39;^#+\s*&#39;, &#39;&#39;, text, flags=re.MULTILINE)<br>    return text</pre><h4>3.3 Stripping HTML Tags &amp; Unescaping Entities</h4><p>Scraped data often contains leftover HTML. First unescape entities (&amp;amp; → &amp;), then remove all tags.</p><pre>import html<br><br>def remove_html_tags(text: str) -&gt; str:<br>    text = html.unescape(text)<br>    return re.sub(r&#39;&lt;[^&gt;]+&gt;&#39;, &#39; &#39;, text)</pre><h4>3.4 Replacing URLs and Emails</h4><p>Instead of deleting URLs (which removes the signal that a link existed), replace them with a placeholder token &lt;URL&gt;. Same for emails.</p><pre>def replace_urls(text: str) -&gt; str:<br>    return re.sub(r&#39;https?://\S+|www\.\S+&#39;, &#39; &lt;URL&gt; &#39;, text)</pre><pre>def replace_emails(text: str) -&gt; str:<br>    return re.sub(r&#39;\b[\w\.-]+@[\w\.-]+\.\w+\b&#39;, &#39; &lt;EMAIL&gt; &#39;, text)</pre><h4>3.5 Unicode Normalization (NFKC)</h4><p>Unicode contains many “compatibility characters” — visually identical glyphs with different code points. NFKC normalizes them to a canonical form.</p><pre>import unicodedata<br>def normalize_unicode(text: str) -&gt; str:<br>    return unicodedata.normalize(&#39;NFKC&#39;, text)</pre><p>Examples: ﬁ → fi, ½ → 1/2, ™ → TM.</p><h4>3.6 Whitespace Cleanup</h4><p>After all the replacements, you’ll have extra spaces, newlines, and tabs. Two functions fix that.</p><pre>def remove_newlines(text: str) -&gt; str:<br>    return text.replace(&#39;\n&#39;, &#39; &#39;).replace(&#39;\r&#39;, &#39; &#39;).replace(&#39;\t&#39;, &#39; &#39;)<br><br>def remove_extra_spaces(text: str) -&gt; str:<br>    return re.sub(r&#39;\s+&#39;, &#39; &#39;, text).strip()</pre><h4>3.7 Reducing Repeated Characters</h4><p>Internet enthusiasm (sooooo goood!!!) creates rare, meaningless tokens. We reduce runs of repeated letters (to at most two) and repeated punctuation to one.</p><pre>def reduce_repeated_letters(text: str) -&gt; str:<br>    return re.sub(r&#39;(.)\1{2,}&#39;, r&#39;\1\1&#39;, text)<br><br>def reduce_repeated_punctuation(text: str) -&gt; str:<br>    return re.sub(r&#39;([!?.])\1+&#39;, r&#39;\1&#39;, text)</pre><h4>3.8 Expanding Contractions and Slang</h4><p>don&#39;t → do not, idk → i do not know, yoe → years of experience. I used a simple dictionary lookup.</p><pre>EXPANSION_MAP = {<br>    &quot;don&#39;t&quot;: &#39;do not&#39;, &quot;can&#39;t&quot;: &#39;cannot&#39;, &quot;i&#39;m&quot;: &#39;i am&#39;,<br>    &#39;idk&#39;: &#39;i do not know&#39;, &#39;tbh&#39;: &#39;to be honest&#39;,<br>    &#39;yoe&#39;: &#39;years of experience&#39;, &#39;atm&#39;: &#39;at the moment&#39;<br>}<br><br>def expand_text(text: str, mapping: dict) -&gt; str:<br>    return &#39; &#39;.join(mapping.get(token.lower(), token) for token in text.split())</pre><h3>4. Tokenisation and Stopwords</h3><p>After cleaning, we split text into tokens. A simple regex works for many English NLP tasks.</p><pre>tokens = re.findall(r&quot;\b\w+\b&quot;, cleaned_text)</pre><p>Stopwords are high‑frequency words like the, is, a, of. Removing them can help focus on content words, but be careful: removing not destroys negation.</p><pre>STOPWORDS = {&#39;the&#39;, &#39;is&#39;, &#39;am&#39;, &#39;are&#39;, &#39;a&#39;, &#39;an&#39;, &#39;in&#39;, &#39;on&#39;, &#39;and&#39;, &#39;of&#39;, &#39;to&#39;, &#39;this&#39;, &#39;that&#39;, &#39;it&#39;, &#39;i&#39;, &#39;with&#39;, &#39;for&#39;, &#39;by&#39;, &#39;from&#39;, &#39;as&#39;, &#39;or&#39;, &#39;but&#39;}<br><br>def remove_stopwords(tokens, stopwords):<br>    return [token for token in tokens if token.lower() not in stopwords]</pre><p>For sentiment analysis, keep stopwords (or at least keep negations). For topic modelling, removal often helps. My pipeline leaves stopwords off by default — you enable them with a flag.</p><h3>5. The Full Configurable Pipeline</h3><p>All the helpers are chained into a single clean_text() function with Boolean flags for each step. This lets you turn operations on/off per task without rewriting code.</p><pre>def clean_text(<br>    text: str,<br>    do_lowercase: bool = True,<br>    do_remove_markdown: bool = True,<br>    do_remove_html: bool = True,<br>    do_replace_urls: bool = True,<br>    do_replace_emails: bool = True,<br>    do_normalize_unicode: bool = True,<br>    do_remove_newlines: bool = True,<br>    do_expand_contractions: bool = True,<br>    do_reduce_repeated: bool = True,<br>    do_remove_stopwords: bool = False,   # off by default<br>    do_remove_numbers: bool = False,     # off by default<br>    do_remove_special: bool = False,     # off by default<br>) -&gt; str:<br>    if not text or not text.strip():<br>        return &#39;&#39;<br>    <br>    if do_remove_markdown:   text = remove_markdown(text)<br>    if do_remove_html:       text = remove_html_tags(text)<br>    if do_replace_urls:      text = replace_urls(text)<br>    if do_replace_emails:    text = replace_emails(text)<br>    if do_normalize_unicode: text = normalize_unicode(text)<br>    if do_remove_newlines:   text = remove_newlines(text)<br>    if do_lowercase:         text = to_lowercase(text)<br>    if do_expand_contractions: text = expand_text(text, EXPANSION_MAP)<br>    if do_reduce_repeated:<br>        text = reduce_repeated_letters(text)<br>        text = reduce_repeated_punctuation(text)<br>    if do_remove_numbers:    text = re.sub(r&#39;\d+&#39;, &#39; &#39;, text)<br>    if do_remove_special:    text = re.sub(r&quot;[^a-z0-9\s.,!?&#39;-]&quot;, &#39; &#39;, text)<br>    <br>    text = remove_extra_spaces(text)<br>    <br>    if do_remove_stopwords:<br>        tokens = re.findall(r&quot;\b\w+\b&quot;, text)<br>        text = &#39; &#39;.join(remove_stopwords(tokens, STOPWORDS))<br>    <br>    return remove_extra_spaces(text)</pre><p>Applying it to the whole corpus:</p><pre>cleaned_posts = [clean_text(text) for text in selftexts]</pre><h3>6. Before vs. After: Real Examples</h3><p>Here’s one of the actual posts before cleaning (first 300 chars):</p><pre>**Client told me MS Copilot replicated what I built. It didn’t.** I built three MVP models for a client over 12 weeks. Nothing fancy: an LSTM, a prophet model, and XGBoost. The difficulty, as usual, was getting and understanding the data and cleaning it...</pre><p>After applying the pipeline (default settings, stopwords not removed):</p><pre>client told me ms copilot replicated what i built. it didn’t. i built three mvp models for a client over 12 weeks. nothing fancy: an lstm, a prophet model, and xgboost. the difficulty, as usual, was getting and understanding the data and cleaning it...</pre><p>Notice:</p><ul><li>Markdown (**bold**) removed.</li><li>Lowercased.</li><li>Domain‑specific terms (LSTM, XGBoost) kept.</li><li>Numbers preserved (12 weeks).</li><li>Punctuation and meaning intact.</li></ul><h3>7. Word Cloud (Visualising the Cleaned Corpus)</h3><p>After cleaning, I generated a word cloud to see what topics dominate r/datascience. This was done after removing stopwords for visual clarity.</p><pre>from wordcloud import WordCloud<br>import matplotlib.pyplot as plt<br><br>wordcloud_text = &#39; &#39;.join(cleaned_tokens)<br>cloud = WordCloud(width=1400, height=700, background_color=&#39;white&#39;).generate(wordcloud_text)<br>plt.imshow(cloud, interpolation=&#39;bilinear&#39;)<br>plt.axis(&#39;off&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmDTuunxgs1_B3_putOziQ.png" /></figure><h3>8. Exporting the Cleaned Dataset</h3><p>The final step is to save the cleaned data alongside the original raw text. I exported a JSON file with this structure:</p><pre>{<br>  &quot;id&quot;: &quot;1ki9zo3&quot;,<br>  &quot;title&quot;: &quot;Client told me MS Copilot replicated what I built. It didn’t.&quot;,<br>  &quot;author&quot;: &quot;melissa_ingle&quot;,<br>  &quot;subreddit&quot;: &quot;datascience&quot;,<br>  &quot;score&quot;: 1,<br>  &quot;raw_selftext&quot;: &quot;... original text ...&quot;,<br>  &quot;clean_selftext&quot;: &quot;... cleaned text ...&quot;<br>}</pre><p>Having both raw and clean versions in the same file means you can always go back and re‑clean with different options without losing the original.</p><h3>9. What’s Next?</h3><p>With a clean corpus, you can now:</p><ul><li>TF‑IDF / BM25 — Build a search engine over the posts.</li><li>Topic modelling (LDA, NMF) — Discover latent themes in r/datascience.</li><li>Sentiment analysis — Classify posts as positive/negative/neutral.</li><li>Embeddings (sentence‑transformers) — Create semantic search.</li><li>Fine‑tune an LLM — Clean training data leads to cleaner generation.</li></ul><p>The complete Jupyter notebook is available on GitHub, so you can run the pipeline on your own Reddit JSON data.</p><h3>Final Thoughts</h3><p>Text cleaning is not glamorous, but it is vital. A model trained on noisy text learns noise. A model trained on clean text learns signal.</p><p>The key lessons:</p><ol><li>Always diagnose first — inspect your corpus before cleaning.</li><li>One function per operation — makes testing and debugging trivial.</li><li>Use flags — keep the pipeline configurable (stopwords off by default!).</li><li>Preserve meaning — replace URLs with placeholders, expand contractions, normalise Unicode.</li><li>Export both raw and clean — because you will change your mind later.</li></ol><p>Now go clean your data. Your models will thank you.</p><p>You can find the full code and the interactive Jupyter notebook on GitHub:[<a href="https://github.com/sokebat/text-cleaning">Click Here</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45134009be46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Curse of Dimensionality: Why Adding More Features Can Break Your Model]]></title>
            <link>https://ashish-subedi.medium.com/the-curse-of-dimensionality-why-adding-more-features-can-break-your-model-512250aed4c6?source=rss-f782bff94d60------2</link>
            <guid isPermaLink="false">https://medium.com/p/512250aed4c6</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ashish Subedi]]></dc:creator>
            <pubDate>Tue, 05 May 2026 13:35:59 GMT</pubDate>
            <atom:updated>2026-05-11T11:39:24.837Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>More data is good. More features? Not always. Here’s why piling on dimensions can quietly destroy your machine learning model — and what to do about it.</blockquote><p>You’re building a model to predict house prices. You start with two features: size and bedrooms. Accuracy is okay. So you add more — age, garage size, roof condition, year renovated…</p><p>Will your model improve? Almost certainly not. In fact, it might get much, much worse.</p><p>Welcome to the <strong>Curse of Dimensionality</strong> — one of the most counter-intuitive traps in machine learning, and one that catches even experienced practitioners off guard.</p><h3><strong>What Is a Dimension?</strong></h3><p>In machine learning, a dimension is just a feature — one column in your dataset. Here’s the tiny dataset we’ll use throughout:</p><pre>import pandas as pd<br><br>houses = pd.DataFrame({<br>    &#39;size_sqft&#39;:   [600, 800, 950, 1100, 1300, 1500, 1700, 1900, 2100, 2400],<br>    &#39;bedrooms&#39;:    [1, 2, 2, 3, 3, 3, 4, 4, 5, 5],<br>    &#39;age_years&#39;:   [30, 10, 25, 5, 15, 2, 8, 20, 3, 12],<br>    &#39;price_$1000&#39;: [120, 180, 165, 240, 220, 310, 330, 290, 420, 480]<br>})</pre><p>We have 3 features → 3 dimensions. <strong>Price is what we want to predict</strong>, not a feature. Now watch what happens when we plot those same 10 houses from 1, 2, and 3 dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WwtDFKtLBJ1eMqNFarVxmw.png" /></figure><p>In 1D the pattern is instant: bigger house = pricier house. In 3D, a 30-year-old house in a great area can cost more than a new house in a bad one — the model has to learn <em>when</em> age matters and when it doesn’t. That’s hard with only 10 examples.</p><blockquote><em>Each new feature adds an axis, and every new axis makes the space the model has to navigate larger — and emptier</em>.</blockquote><h3><strong>The Curse in Action</strong></h3><p>Here’s the mathematical core of the curse. Imagine dividing each feature into 10 bins (small buckets). How much of the grid does our data actually cover?</p><pre>1 feature  →  10 boxes    →  10 houses cover  100.0%<br>2 features →  100 boxes   →  10 houses cover   10.0%<br>3 features →  1,000 boxes →  10 houses cover    1.0%<br>4 features →  10,000 boxes→  10 houses cover    0.1%<br>6 features →  1,000,000  →  10 houses cover   0.001%</pre><p>The number of boxes grows as 10ᵈⁱᵐᵉⁿˢⁱᵒⁿˢ. Your data stays at 10 houses. Result: your model is operating in a nearly empty universe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bracrHC4PRXj6CV2DNSQgg.png" /></figure><blockquote><em>Left: coverage collapses to nothing. Right: space grows exponentially .</em></blockquote><h3>How It Breaks Your Actual Model</h3><p>This isn’t just abstract math. Let’s make it concrete by simulating what happens when we add features one at a time and track the prediction error at each step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hJtlAFtwhRvgjAyIhDlSkA.png" /></figure><p>The sweet spot here is 3 features — size, bedrooms, and bathrooms — with a $14k average error. After that, every new feature makes the model worse. Not because the features are <em>wrong</em>, but because our 10-house dataset can’t fill the expanding space they create.</p><blockquote><strong>Rule of thumb:</strong> For every new feature you add, you need roughly <strong>10× more data</strong> to maintain the same model quality. Add 3 features → you need 1,000× more data. That’s rarely available.</blockquote><h3>Diagnose With Correlations</h3><p>Before reaching for a fix, check your correlation matrix. This is the single most important diagnostic for whether your dataset is suffering from dimensional bloat.</p><pre>import seaborn as sns<br><br>corr = houses.corr()<br>sns.heatmap(corr, annot=True, fmt=&#39;.2f&#39;, cmap=&#39;RdYlBu_r&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*u5tbgdRhVsP2DiKHCCcGjQ.png" /></figure><p>size_sqft vs bedrooms: correlation is <strong>~0.99</strong>. These two features are near-perfect duplicates. Keeping both is like hiring two employees to do the exact same job — you pay double the cost and get zero extra work done. Meanwhile age_years has low correlation with the others — it&#39;s carrying genuinely independent information.</p><h3>The Fix: PCA (Principal Component Analysis)</h3><p>The fix isn’t always to collect more data. Often it’s to be smarter about which dimensions you use.</p><p>First, a quick check: in our dataset, size_sqft and bedrooms are almost perfectly correlated . Keeping both is like having two employees doing the exact same job.</p><p>PCA combines correlated features into new “super‑features” called components. Think of it like summarizing a 500‑page book into 3 key paragraphs — you lose a little detail, but gain something far more compact and useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMOI0F1uDqlJehBha5wnOw.png" /></figure><blockquote><em>Left: size vs bedrooms plotted together — data forms a diagonal line (redundancy).</em></blockquote><blockquote><em>Right: after PCA, Component 1 captures 99% of the information. Component 2 (vertical axis) is nearly flat — it adds almost nothing.</em></blockquote><p>We went from <strong>2 features → 1 component</strong> and kept 99% of the information. Our model now works in a lower-dimensional space — meaning our 10 houses cover a much larger fraction of it, and predictions improve.</p><h3>Conclusion</h3><p>When I first heard “the curse of dimensionality” it sounded like an academic warning — something that only matters at scale. What building this from scratch with 10 houses taught me is that the math starts working against you at 3 or 4 features with small datasets. The space doesn’t just grow — it <em>explodes</em>, leaving your model navigating a universe it has almost no examples to learn from.</p><p>The correlation heatmap is your early-warning system. PCA is your surgical fix. Feature engineering — the deliberate work of choosing, combining, and trimming — is where most of the real leverage lives in machine learning.</p><p>The visualizations above come from the accompanying notebook. You can run it yourself to explore the curse of dimensionality interactively.<br> [<a href="https://github.com/sokebat/curse-of-dimensionality">Github</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=512250aed4c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nepal Customs Trade Analysis Report (2020/21–2024/25)]]></title>
            <link>https://ashish-subedi.medium.com/nepal-customs-trade-analysis-report-2020-21-2024-25-812af9618652?source=rss-f782bff94d60------2</link>
            <guid isPermaLink="false">https://medium.com/p/812af9618652</guid>
            <category><![CDATA[nepal]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[nepal-custom-trade]]></category>
            <dc:creator><![CDATA[Ashish Subedi]]></dc:creator>
            <pubDate>Sun, 22 Mar 2026 12:02:47 GMT</pubDate>
            <atom:updated>2026-03-22T12:08:04.921Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>This report examines the trends in imports and exports across Nepal’s major customs points over the past five fiscal years (2020/21–2024/25). It provides a detailed look at how trade is distributed among different customs, highlights the balance between imports and exports at key border points, and explores the emerging potential of northern trade routes. Additionally, the report analyzes patterns of trade concentration to identify key hubs and potential risks. The insights presented are based on official customs trade data, offering a clear and data-driven perspective on Nepal’s cross-border trade dynamics.</p><h3>Overview</h3><p>To understand Nepal’s import patterns, we analyzed total import volumes across all customs points for the last five fiscal years (2020/21–2024/25). This allowed us to identify which customs serve as the primary gateways for imported goods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88xz2PCmXHqe4lxq5Z6aBQ.png" /></figure><h4>Top 3 Trade Share:</h4><p>● <strong>Birgunj Customs Office (35.0%):</strong> Clearly the leading trade hub, Birgunj accounts for over a third of Nepal’s total trade, highlighting its vital role as the country’s primary southern gateway.</p><p>● <strong>Bhairahawa Customs Office (15.2%):</strong> Positioned second, Bhairahawa handles a substantial share of trade, serving as an important secondary route that complements the main southern corridor.</p><p>● <strong>Biratnagar Customs Office (12.8%):</strong> Ranking third, Biratnagar plays a key role in facilitating trade through eastern routes, reflecting its significance in supporting regional commerce.</p><h3>Import vs Export Distribution (Top 3 Custom Points)</h3><p>To gain deeper insights into Nepal’s trade dynamics and better understand the flow of goods across the country, we analyzed the balance between imports and exports for the top three customs points that handle the largest volumes of imported goods</p><h4>Birgunj Customs Office</h4><p>At Birgunj Customs Office, imports overwhelmingly dominate trade, accounting for 90.8% of the total volume, while exports contribute only 9.2% Trade at the Birgunj Customs Office is overwhelmingly inbound, with imports far exceeding exports. For every NPR 1 worth of goods exported, roughly NPR 10 worth of goods are imported</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*Rtpsgv87ifbvkcp2Sj4Bzg.png" /></figure><h4>Bhairahawa Customs Office</h4><p>At Bhairahawa Customs Office, imports overwhelmingly dominate trade, accounting for 95.8% of the total volume, while exports contribute only 4.2%. Trade at this hub is almost entirely inbound, making it the most import-heavy among the major customs points. For every NPR 1 worth of goods exported, roughly NPR 23 worth of goods are imported</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*aJMsjt9SJ10D0LkAAqaPlQ.png" /></figure><h4>Biratnagar Customs Office</h4><p>At Bhairahawa Customs Office, imports overwhelmingly dominate trade, accounting for 95.8% of the total volume, while exports contribute only 4.2% Trade at this hub is still largely inbound, but the export component is noticeably higher than at Birgunj. For every NPR 1 worth of goods exported, approximately NPR 4 worth of goods are imported</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*UkShEOgTMBb6DdbxAgpNbQ.png" /></figure><h3>Emerging Northern Trade Routes (Imports)</h3><p>To explore the potential of northern trade corridors, we focused on Rasuwa and Tatopani customs points. Only import volumes were analyzed to track trends in cross-border trade, and data was examined over the past five fiscal years (2020/21–2024/25).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*ub1va28T8qxO0s4AStWPUQ.png" /></figure><h4>Observations:</h4><p>● Rasuwa Customs Office demonstrates a steady upward trend in imports, suggesting gradual development of northern trade routes and growing engagement with cross-border trade.</p><p>● Tatopani Customs Office shows fluctuating import patterns, likely influenced by changes in cross-border policies or periodic infrastructure constraints.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=812af9618652" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>