<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anuj Goel on Medium]]></title>
        <description><![CDATA[Stories by Anuj Goel on Medium]]></description>
        <link>https://medium.com/@goelanuj132.in?source=rss-4f4f4b9f9cbe------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*LBRVyFPEhutxUkQY</url>
            <title>Stories by Anuj Goel on Medium</title>
            <link>https://medium.com/@goelanuj132.in?source=rss-4f4f4b9f9cbe------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:26:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@goelanuj132.in/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Recommender Systems Aren’t Prediction Problems — They’re Causal Problems]]></title>
            <link>https://medium.com/@goelanuj132.in/recommender-systems-arent-prediction-problems-they-re-causal-problems-c8307448f62d?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8307448f62d</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[econometrics]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 17:58:08 GMT</pubDate>
            <atom:updated>2026-02-15T17:58:08.416Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why “What will the user click?” is the wrong question, and “What happens because we recommended it?” is the right one.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vOq1oF_KOc-_NX0VmMKmew.jpeg" /></figure><h3>Introduction: The Prediction Trap</h3><p>In the data science community, Recommender Systems are often treated as the ultimate prediction challenge. The industry standard — from collaborative filtering to modern deep learning candidates — focuses heavily on one metric: accuracy. The goal is to predict, with the highest possible probability, what a user will click on or purchase.</p><p>The logic appears sound. If a model can accurately predict that a user prefers Item B over Item A, recommending Item B seems like the correct decision.</p><p>However, this “prediction-first” mindset overlooks a subtle but critical flaw in how we define value. Susan Athey, the Economics of Technology Professor at Stanford and a leading voice in Causal Inference, highlighted this distinction during a lecture for the Becker Friedman Institute:</p><blockquote>“Recommendations are considered a prediction problem when in fact any kind of objective that you would want to use recommendations for is actually a causal problem.”</blockquote><h4>The Accuracy Illusion</h4><p>To understand the weight of Athey’s statement, we must look at the objective functions that drive most commercial systems.</p><p>In a standard supervised learning setup, we train models to estimate a conditional probability:</p><blockquote><strong><em>P(Y | X)</em></strong></blockquote><p>Where:</p><ul><li><strong>Y</strong> is the outcome (e.g., a purchase or click).</li><li><strong>X</strong> represents the context (user features, history, etc.).</li></ul><p>Models are trained on historical data to find correlations. If the data shows that users who buy high-end gaming laptops also tend to buy high-performance mice, the model calculates a high probability — <strong>P(Y=1 | X)</strong> — and recommends the mouse.</p><p>However, prediction is passive. It observes the world as it exists.</p><h4>The Business Objective</h4><p>Businesses, conversely, are not passive observers. When an organization deploys a recommendation engine, the goal is rarely just to <em>predict</em> a purchase. The goal is to <em>influence</em> it.</p><p>The business question is not, &quot;What is the user likely to buy?&quot; The true business question is, &quot;How can I influence the user to buy something they <strong>would not have bought otherwise</strong>?&quot;</p><p>This shifts the problem from <strong>Prediction</strong> to <strong>Intervention</strong>.</p><p>We are no longer trying to estimate <strong>P(Y|X)</strong>. We are trying to estimate the effect of an action <strong>T</strong> (the recommendation) on the outcome <strong>Y</strong>. In the language of Causal Inference, this is the <strong>Treatment Effect</strong>.</p><p>When we confuse these two objectives, we risk optimizing for what is known as &quot;Organic Utility&quot;—recommending items to users who were already planning to buy them. The model claims credit for the sale (high accuracy), but the actual incremental value to the business is negligible.</p><p>To build truly effective systems, we must stop treating recommendations as passive predictions and start treating them as active interventions.</p><h3>The Intuitive Failure: The “Harry Potter” Problem</h3><p>To understand why high-accuracy models fail to generate value, let’s look at a classic industry scenario often called the “Harry Potter Problem.”</p><p>Imagine you are building a recommendation engine for a bookstore during the height of the Harry Potter craze. Every third customer walking through the door is there specifically to buy <em>Harry Potter and the Deathly Hallows</em>.</p><p>Your model, trained to maximize purchase probability <strong>P(Y | X)</strong>, notices this trend immediately. It learns that regardless of age, gender, or past history, the probability of a user buying this book is effectively 100%.</p><p>Consequently, the model recommends Harry Potter to everyone.</p><h4>The Illusion of Success</h4><p>If you evaluate this model using standard metrics — Accuracy, Precision, or Recall — it looks like a masterpiece.</p><ul><li><strong>Did the user buy the book?</strong> Yes.</li><li><strong>Did the model recommend it?</strong> Yes.</li><li><strong>Accuracy:</strong> 100%.</li></ul><p>But from a business perspective, the model has failed.</p><p>The users were going to buy the book anyway. The recommendation did not <em>cause</em> the purchase; it merely <em>coincided</em> with it. In causal terms, the <strong>Treatment Effect</strong> was zero. The recommendation wasted valuable screen real estate on a “Sure Thing” rather than exposing the user to a niche title they might have missed (a “Persuadable”).</p><h4>The Four Types of Users</h4><p>This distinction brings us to the most important concept in Causal Recommendation: <strong>Uplift Modeling</strong>.</p><p>When we move from prediction to causal intervention, we are not just looking at who buys. We are looking at how the <em>recommendation</em> changes their behavior. This segments our users into four distinct quadrants:</p><p>1.<strong> The Sure Things:</strong> Users who will buy the item whether you recommend it or not (e.g., the Harry Potter fans).</p><ul><li><em>Action:</em> <strong>Do not disturb.</strong> Don’t waste your budget or screen space here.</li></ul><p>2. <strong>The Lost Causes:</strong> Users who will never buy the item, regardless of your recommendation.</p><ul><li><em>Action:</em> <strong>Do not disturb.</strong></li></ul><p>3.<strong> The Do-Not-Disturbs (Sleeping Dogs):</strong> Users who would have stayed (or bought) if left alone, but leave (or don’t buy) if you annoy them with a recommendation.</p><ul><li><em>Action:</em> <strong>Avoid actively.</strong></li></ul><p>4.<strong> The Persuadables:</strong> Users who will buy the item <em>only if</em> you recommend it.</p><ul><li><em>Action:</em> <strong>Target Aggressively.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hkWwV3UOenK-iqYHkbQIg.png" /></figure><h4>The “Prediction” Trap Revisited</h4><p>Standard predictive models (looking at <strong>P(Y | X)</strong>) cannot distinguish between a <strong>Sure Thing</strong> and a <strong>Persuadable</strong>. They lump them together because both groups result in a sale (<strong>Y=1</strong>).</p><p>A causal model, however, aims to identify the <strong>Persuadables</strong>. It optimizes for the <em>difference</em> in behavior, not the behavior itself.</p><p>If your recommender system is purely predictive, it is likely over-indexing on <strong>Sure Things</strong> — cannibalizing your organic sales while claiming credit for revenue that would have happened anyway.</p><h3>The Mathematics of Intervention</h3><p>To solve the “Harry Potter” problem, we need to move beyond intuition and look at the mathematical objective functions we are optimizing.</p><p>In a standard Recommender System, our data consists of user interactions. We can formalize this setup using the potential outcomes framework (Rubin Causal Model).</p><p>Let’s define our variables:</p><ul><li><strong>Y</strong>: The Outcome (e.g., Purchase = 1, No Purchase = 0).</li><li><strong>T</strong>: The Treatment (Recommendation shown = 1, Not shown = 0).</li><li><strong>X</strong>: The User Context (Demographics, history, device).</li></ul><h4><strong>The Prediction Equation (What we usually do)</strong></h4><p>When we train a standard supervised learning model (like Matrix Factorization or a Deep Neural Network), we are minimizing the error for the following conditional expectation:</p><blockquote><strong><em>E[Y | X, T=1]</em></strong></blockquote><p>Translated, this asks: <em>“Given that we showed the recommendation to User X, what is the probability they purchased the item?”</em></p><p>This equation has a critical blind spot. It ignores the counterfactual scenario. It assumes that if <strong>Y=1</strong>, the recommendation (T=1) deserves the credit. It does not ask what would have happened if we <em>hadn’t</em> shown the recommendation.</p><h4><strong>The Causal Objective (What we should do)</strong></h4><p>To optimize for <strong>Uplift</strong> (incremental value), we need to calculate the difference between two potential worlds. We are interested in the <strong>Individual Treatment Effect (ITE)</strong>, often denoted as <strong>τ(x)</strong>:</p><blockquote><strong><em>τ(x) = E[Y | X, T=1] − E[Y | X, T=0]</em></strong></blockquote><p>This equation asks a fundamentally different question: <em>“How much </em><strong><em>more</em></strong><em> likely is User X to purchase the item because we recommended it, compared to if we had done nothing?”</em></p><ul><li><strong>E[Y | X, T=1]:</strong> The probability of purchase <em>with</em> the recommendation.</li><li><strong>E[Y | X, T=0]:</strong> The probability of purchase <em>without</em> the recommendation.</li></ul><p>If a user is a “Sure Thing” (like a Harry Potter fan), both values are high (e.g., 0.95–0.90), resulting in an uplift of near zero. If a user is a “Persuadable,” the difference is large (e.g., 0.60–0.10), resulting in a high uplift.</p><h4>The Fundamental Problem of Causal Inference</h4><p>Here lies the central challenge of Causal Inference. In the real world, we can never observe both potential outcomes for the same user at the same time.</p><p>For any specific user at a specific moment, we either show the recommendation (<strong>T=1</strong>) or we don’t (<strong>T=0</strong>). We see one outcome, and the other becomes the <strong>counterfactual</strong> — the road not taken.</p><p>This is why standard supervised learning fails here. In a typical prediction task (like classifying images of cats vs. dogs), we have the ground truth labels. In causal inference, we are missing half the labels. We never see the counterfactual, so we cannot calculate the error for <strong>τ(x)</strong> directly.</p><p>This “missing data” problem forces us to rely on specific techniques — like Randomized Control Trials (A/B Tests) or specialized Causal ML models (S-Learners, T-Learners) — to estimate that missing counterfactual.</p><h3>Confounders &amp; Feedback Loops: Why Your Data is Lying to You</h3><p>If we accept that we need to measure the <strong>Treatment Effect</strong> (uplift), the next logical step is to look at our historical data. We have logs of millions of user interactions. Surely we can just compare users who saw a recommendation against those who didn’t?</p><p>Unfortunately, this approach is often doomed to fail due to <strong>Confounding Variables</strong>.</p><p>In a perfect world (like a Randomized Control Trial), the decision to show a recommendation (T) is random. In the real world, it is not. Our recommendation engines are designed to be <em>biased</em> — they show specific items to specific people based on rules or previous models.</p><h4>The Hidden Third Variable</h4><p>Imagine you are analyzing the performance of a “Luxury Watch” recommendation.</p><ul><li><strong>Group A (Saw Rec):</strong> High conversion rate.</li><li><strong>Group B (Didn’t See Rec):</strong> Low conversion rate.</li></ul><p>A naive model concludes: <em>“The recommendation caused the purchase!”</em></p><p>But what if the recommendation engine only showed luxury watches to users with <strong>High Income</strong> (Z)?</p><ul><li><strong>High Income Users (Z=1):</strong> More likely to see the recommendation (T=1).</li><li><strong>High Income Users (Z=1):</strong> More likely to buy the watch regardless (Y=1).</li></ul><p>In this scenario, <strong>Income</strong> is a <strong>Confounder</strong>. It influences both the treatment and the outcome. The correlation we see between the recommendation and the purchase is spurious; it’s actually just a reflection of the user’s wealth.</p><p>If we don’t control for Z, our model will overestimate the value of the recommendation. It will attribute the sales to the algorithm, when in reality, it was just demographics.</p><h4>The Feedback Loop (The “Echo Chamber”)</h4><p>This problem gets worse when we deploy the model.</p><ol><li><strong>Step 1:</strong> The model (mistakenly) learns that “Rich People + Luxury Watch Recommendation = Sale.”</li><li><strong>Step 2:</strong> It aggressively targets <em>only</em> rich people with luxury watch ads.</li><li><strong>Step 3:</strong> The training data for the <em>next</em> version of the model now contains <em>only</em> rich people seeing the ad.</li></ol><p>We have created a <strong>Feedback Loop</strong>. The model has blinded itself to the counterfactual (what happens if we show the watch to a middle-income user?). It effectively “censors” its own training data, reinforcing its initial biases and preventing it from ever learning the true causal effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dV19TnBObe0dGeokw-P94g.png" /></figure><h4>Simpson’s Paradox</h4><p>This can lead to a statistical phenomenon known as <strong>Simpson’s Paradox</strong>, where a trend appears in different groups of data but disappears or reverses when these groups are combined.</p><p>You might find that your recommendation increases sales overall (when looking at the whole population), but when you split by income brackets, it actually <em>decreases</em> sales in every single bracket. The aggregated data lies because the <em>mix</em> of users receiving the recommendation is unbalanced.</p><h3>Technical Solution: From Predicting to Uplifting</h3><p>So, how do we fix this? If our historical data is biased and our standard predictive models are optimizing for the wrong metric, what is the alternative?</p><p>The solution lies in a specific branch of machine learning known as <strong>Uplift Modeling</strong> (or Causal Machine Learning).</p><h4>The Gold Standard: Randomized Control Trials (RCTs)</h4><p>The most robust way to break the feedback loop is to generate unbiased data. This is often done through <strong>A/B Testing</strong> or <strong>Randomized Control Trials (RCTs)</strong>.</p><p>In an RCT, we randomly assign users to a <strong>Treatment Group</strong> (Show Recommendation) or a <strong>Control Group</strong> (Do Not Show Recommendation). By randomizing <strong>T</strong>, we break the correlation between user features (<strong>X</strong>) and the treatment. This ensures that any difference in outcome (<strong>Y</strong>) is actually caused by the recommendation, not by confounders like income or popularity.</p><p>Once we have this unbiased data, we can train models to estimate the <strong>Conditional Average Treatment Effect (CATE)</strong>.</p><h4>Modeling Approaches: S-Learners and T-Learners</h4><p>There are two common meta-learners used to estimate uplift:</p><p><strong>1. The S-Learner (Single Model)</strong> We train a single predictive model that includes the treatment indicator <strong>T</strong> as a feature alongside user features <strong>X</strong>.</p><ul><li><strong>Model:</strong> Y ~ f(X, T)</li><li><strong>Prediction:</strong> We predict the outcome twice for every user: once assuming <strong>T=1</strong> and once assuming <strong>T=0</strong>.</li><li><strong>Uplift Calculation:</strong> τ(x) = f(X, 1) − f(X, 0)</li></ul><p><strong>2. The T-Learner (Two Models)</strong> We train two separate models: one on the treated group and one on the control group.</p><ul><li><strong>Model 1 (Treated):</strong> μ1(X) = E[Y | X, T=1]</li><li><strong>Model 2 (Control):</strong> μ0(X) = E[Y | X, T=0]</li><li><strong>Uplift Calculation:</strong> τ(x) = μ1(X) − μ0(X)</li></ul><h4>The Objective Function Shift</h4><p>The most critical technical shift happens in the loss function.</p><ul><li><strong>Traditional RecSys:</strong> Minimizes error on the <strong>Outcome</strong> (Y).</li><li><em>Goal:</em> Minimize (Y_actual - Y_predicted)²</li><li><strong>Uplift Modeling:</strong> Minimizes error on the <strong>Effect</strong> (τ).</li><li><em>Goal:</em> We cannot observe τ directly, so we often use proxy metrics like <strong>Transformed Outcome Loss</strong> or optimize for <strong>Area Under the Uplift Curve (AUUC)</strong> rather than AUC-ROC.</li></ul><p>By changing the mathematical objective, the model stops caring about “Who buys?” and starts caring about “Who is persuadable?”</p><h3>6. Conclusion: The Paradigm Shift</h3><p>Susan Athey’s observation — that recommendations are causal problems disguised as prediction problems — is more than just academic theory. It is a fundamental shift in how we design intelligent systems.</p><p>As Data Scientists, we are trained to chase accuracy. We celebrate when our model achieves a high AUC or low RMSE. But in the context of recommendations, accuracy can be misleading. A model that perfectly predicts organic behavior provides <strong>zero incremental value</strong> to the business. In fact, it often destroys value by giving away discounts or prime screen real estate to users who were already converted.</p><p>True value comes from identifying the <strong>Persuadables</strong> — the users who are on the fence, waiting for a nudge.</p><p>Moving from <strong>Prediction (What will happen?)</strong> to <strong>Causal Inference (What happens if I do this?)</strong> is not just a technical upgrade; it is a strategic necessity. As AI systems become more autonomous, the ability to distinguish between correlation and causation will define the next generation of effective, intelligent products.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8307448f62d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Data Pipeline as a Single System, Not Independent Steps]]></title>
            <link>https://medium.com/@goelanuj132.in/the-data-pipeline-as-a-single-system-not-independent-steps-1868d4aedb87?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/1868d4aedb87</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Thu, 08 Jan 2026 13:19:41 GMT</pubDate>
            <atom:updated>2026-01-08T13:19:41.938Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Why treating data science as an assembly line causes silent failures — and what a real system looks like instead</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RujgAXWB90z00se5tkfXkA.jpeg" /></figure><p>If you open almost any introductory textbook on Data Science, or enroll in a standard Machine Learning MOOC, you are almost immediately presented with a comforting diagram. It usually looks like a flowchart. On the left, a cylinder labeled <em>Raw Data</em>. An arrow points to <em>Preprocessing</em>, which points to <em>EDA</em>, then <em>Training</em>, and finally, at the far right, <em>Evaluation</em>.</p><p>It is clean.<br> It is linear.<br> It resembles an industrial assembly line.</p><p>In this model, data science is portrayed as a series of independent, discrete stations. Once a stage is completed, it is “done.” You clean the data, save it as cleaned_data.csv, and move on. You generate features, save them as X_train.npy, and move on. You train the model, save a .pkl file, and you are finished. The implicit promise of this workflow is modularity: you can tweak the preprocessing step without fundamentally breaking evaluation, as long as the shapes line up.</p><p>This promise is false.</p><p>The “assembly line” model is not malicious — it is pedagogically convenient. It works in classrooms because the dataset is static. It fails in the real world because data is not a frozen artifact. In production systems, data is a living, turbulent stream. When we treat a data pipeline as a sequence of isolated steps, we fall into a trap of <strong>false decoupling</strong>. We believe we are assembling Lego blocks that can be swapped freely, when in reality we are stacking a Jenga tower. Pull a block near the bottom — a small change in data cleaning — and the collapse happens at the top, often silently.</p><h3>How tools train us to think linearly</h3><p>The assembly line mindset is reinforced by the tool most data scientists rely on: the Jupyter Notebook.</p><p>The notebook’s structure enforces linear thinking. Code is written in cells. We run imports, then load data, then clean it. As we scroll down, we mentally check off tasks: <em>null values handled</em>, <em>outliers removed</em>, <em>features ready</em>. Once a section scrolls past the viewport, it leaves our working memory. Data cleaning stops being active logic and becomes a completed chore.</p><p>From a software engineering perspective, this is a nightmare of <strong>global state</strong> and <strong>hidden dependencies</strong>. DataFrames are mutated in place. Cells are re-run out of order. Execution history matters as much as the code itself. Two notebooks that look identical can produce different results depending on what ran earlier and what did not.</p><p>More importantly, notebook-driven workflows create a conceptual separation between the <em>logic of the data</em> and the <em>logic of the model</em>. The model — the algorithm, the hyperparameters, the architecture — is treated as the “smart” part of the system. The pipeline that feeds it is treated as plumbing.</p><p>This separation is artificial.</p><p>In a real system, the imputation strategy <em>is part of the model</em>. Change how missing values are handled, and you have mathematically altered the hypothesis space the model is capable of learning. Encoding choices reshape geometry. Scaling choices redefine distance. Treating these as steps you “finish” before modeling begins obscures the fact that they are inseparable from the model itself.</p><h3>Tight coupling is not a flaw — it is the nature of data systems</h3><p>In software engineering, we strive for loose coupling. Modules expose contracts. Internal changes do not ripple outward.</p><p>Data pipelines violate this principle by definition.</p><p>Every stage in a pipeline changes not just the data, but the <em>meaning</em> of the data. Scaling defines what distance means. Encoding defines what similarity means. Aggregation defines temporal resolution. These choices cannot be isolated. Treating them as independent steps creates a false sense of modularity.</p><p>The coupling is usually invisible because code still runs. Interfaces still match. Arrays still have the right shape. But correctness in data systems is not defined by successful execution — it is defined by <strong>semantic consistency</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VmdH7zzpZ2E3EJrEGZU_5g.png" /></figure><h3>Silent failure: when nothing crashes, but everything is wrong</h3><p>Traditional software fails loudly. Break a dependency and the system throws an exception. The assembly line stops and alarms go off.</p><p>Machine learning systems fail quietly.</p><p>A change in upstream data semantics may not break a single line of code. The pipeline executes. The model returns predictions. The API responds with 200 OK. But the system has become wrong.</p><p>Silent failure is the most dangerous failure mode in data systems because it produces confident output without correctness. The assembly line model optimizes for syntactic validity, not semantic stability.</p><h3>When isolated “steps” break down in practice</h3><p>The assembly line mindset is not just conceptually flawed; it is one of the primary sources of failure in production machine learning.</p><p>One common failure mode is <strong>training–serving skew</strong>. When preprocessing logic exists as instructions rather than as a compiled artifact, it must be reinterpreted across environments. Slight differences in numerical behavior, defaults, or execution context produce input vectors that are mathematically different from those seen during training. The model predicts confidently — but on a world it was never trained to understand.</p><p>Another failure mode is <strong>data leakage</strong>. In classrooms, leakage is taught as a careless mistake. In real systems, it is almost always architectural. Operations like normalization or encoding depend on <em>when</em> and <em>from which subset</em> statistics are computed. When preprocessing is treated as a dataset-wide operation rather than as a context-aware transformation, information flows backward across boundaries that were assumed to be isolated. Evaluation looks excellent in the lab and collapses in production.</p><p>A third failure mode is the accumulation of <strong>technical debt</strong>. Decoupled steps invite glue code. Scripts multiply. Data is exported, re-imported, and transformed repeatedly across teams. Over time, the pipeline becomes a jungle of undocumented logic. No single component is broken, but no one understands the system end-to-end. Innovation slows because change becomes dangerous.</p><h3>The missing layer: semantic and statistical contracts</h3><p>Most pipelines define schemas. Very few define meaning.</p><p>A column can be present, correctly typed, and within expected ranges — yet represent something different from what downstream logic assumes. Schema validation catches shape errors. It does not catch semantic drift.</p><p>A robust data system requires <strong>contracts</strong>, not just schemas.</p><p>Semantic contracts answer questions like:<br> What does a missing value represent?<br> What distribution was this transformation fit on?<br> What population does this data describe?<br> What temporal assumptions underlie this aggregation?</p><p>Statistical contracts define invariants: expected ranges, distributional properties, correlations that should not silently change. Without these contracts, systems cannot detect meaning drift. They can only detect crashes. Silent failure becomes inevitable.</p><h3>The shift that matters: the pipeline is the model</h3><p>To a junior data scientist, the model is the final algorithm — the Random Forest, the XGBoost regressor, the neural network.</p><p>To an architect, the model is the <strong>entire transformation graph</strong>, from raw input to prediction.</p><p>Preprocessing parameters are hyperparameters. Encoders carry learned state. Thresholds and vocabularies are part of the hypothesis. Saving only model weights is incomplete serialization.</p><p>This is where software engineering principles become essential.</p><p>In well-designed systems, data and the logic that operates on it are encapsulated together. In data science, we often violate this immediately by scattering logic across scripts and artifacts.</p><p>The correct abstraction is a single pipeline object with a unified interface. A .fit() method defines the world the model learns from. A .predict() method enforces that same world at inference. Preprocessing is no longer a preparatory script; it is a compiled, versioned component of the model.</p><p>This is not a tooling preference. It is a correctness requirement.</p><h3>From scripts to systems</h3><p>When the pipeline is treated as a first-class object, several things change.</p><p>Preprocessing logic is no longer reimplemented — it is reused.<br>Training and inference share the same transformation graph.<br>State is explicit and serializable.<br>Assumptions are localized rather than scattered.</p><p>Most importantly, responsibility becomes clear. There is a single source of truth for how raw data becomes a prediction.</p><h3>Closing the loop: from pipelines to living systems</h3><p>Unlike traditional software, machine learning systems interact with a changing world. Data shifts even when code does not. A system that ends at deployment is incomplete.</p><p>A production-grade system closes the loop.</p><p>It versions data alongside code, so failures can be reproduced rather than guessed at. It monitors <strong>statistical health</strong> — distributions, ranges, correlations — not just latency and error rates. It treats drift as a first-class signal rather than an afterthought.</p><p>Retraining is not a manual ritual triggered by intuition. It is a system response to detected change. New data triggers evaluation. Evaluation triggers comparison. Comparison triggers promotion or rollback. Human review exists where it matters, but the system itself is aware that the world is moving. This is the difference between a project and a product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MRV4I3ZElqVrDXkjrUD_DQ.png" /></figure><h3>The architect’s view</h3><p>The data pipeline is not a list of steps.</p><p>It is a system.</p><p>Code shapes the pipeline.<br>The pipeline shapes the model.<br>The model produces predictions.<br>Predictions generate new data.<br>Data reveals drift.<br>Drift demands adaptation.</p><p>This cycle is not advanced architecture. It is the <strong>minimum viable architecture</strong> for reliable machine learning.</p><p>Treat the pipeline as plumbing, and the system will fail silently.<br>Treat it as the system itself, and intelligence becomes sustainable.</p><blockquote><strong>In practice, most machine learning failures are not caused by bad models, but by systems that never understood themselves.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1868d4aedb87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Data Preprocessing Is an Epistemological Problem, Not a Technical One]]></title>
            <link>https://medium.com/@goelanuj132.in/why-data-preprocessing-is-an-epistemological-problem-not-a-technical-one-d4740d50675b?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4740d50675b</guid>
            <category><![CDATA[epistemology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-bias]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Sat, 03 Jan 2026 19:59:19 GMT</pubDate>
            <atom:updated>2026-01-03T19:59:19.510Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This essay reflects a conceptual perspective on data preprocessing; examples are illustrative rather than exhaustive.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IlAOVi96cYrsrJIEWzeQtA.jpeg" /></figure><p>Data preprocessing is often described as a technical necessity — the mundane work of fixing typos, formatting dates, and filling in blanks so that an algorithm does not crash. This framing treats preprocessing as a preliminary inconvenience, a janitorial task that precedes “real” analysis.</p><p>That framing is deeply misleading.</p><p>Every preprocessing step is a decision about what counts as data, what differences matter, and which variations are safe to ignore. Those decisions shape what can be known from a dataset and what must remain invisible. In this sense, data preprocessing is not merely technical work; it is an epistemological act. It determines the conditions under which knowledge is constructed.</p><p>Epistemology concerns how knowledge is formed, justified, and limited. When applied to data science, it asks a simple but uncomfortable question: <em>what kind of world must exist for this dataset to make sense?</em> Preprocessing is where that world is quietly built.</p><h3><strong>The Construction of Categories: Ontology Before Prediction</strong></h3><p>A large portion of preprocessing involves categorization: binning values, grouping labels, encoding concepts into manageable forms. Technically, this is described as dimensionality reduction or feature simplification. Epistemologically, it is an act of ontology — the declaration of what kinds of things exist in the model’s universe.</p><p>Consider a dataset of mental health journals. One entry describes the sharp, acute distress of failing an exam — sadness. Another describes the heavy, persistent fog of losing a parent — grief. If these are grouped into a single label such as <em>negative emotion</em> to simplify classification, a profound transformation occurs.</p><p>From a technical perspective, the model now has fewer classes and converges more easily. From an epistemological perspective, the model now inhabits a world where sadness and grief are indistinguishable. Duration, depth, and existential weight cease to exist as meaningful dimensions. The model learns that a bad grade and a funeral are functionally equivalent triggers of “unhappiness.”</p><p>The consequence is not merely loss of nuance. It is the creation of a false equivalence that the model treats as absolute truth. Any recommendation or inference made downstream — perhaps suggesting a short walk or a breathing exercise — will be equally banal for both situations. The model cannot recognize what it was never told exists.</p><p>This is not an error in modeling. It is a consequence of category construction.</p><h3><strong>Taxonomies as Frozen Stories</strong></h3><p>Classification systems are often treated as neutral mirrors of reality. In practice, they are historical artifacts. Taxonomies encode the assumptions, priorities, and power structures of the moment in which they were created.</p><p>Sociologists of classification have described taxonomies as “frozen stories”: narratives about the world that have been stabilized and operationalized. Once frozen, they resist change — even when reality shifts.</p><p>Historical census data provides a clear example. Categories such as “Mulatto” or “Octoroon” appeared in nineteenth-century demographic records and later disappeared as social understandings of race evolved. A model trained on such data without careful preprocessing inherits an ontology that no longer exists. Worse, it may attempt to force contemporary individuals into obsolete categories simply because those are the categories the dataset provides.</p><p>A more common modern example is the ubiquitous <em>Other</em> bucket. To address high cardinality, rare categories are collapsed into a single residual class. Technically, this improves stability. Epistemologically, it is an act of silencing. The model is told that everything in this bucket lacks distinct identity or explanatory importance. When marginalized groups or rare medical conditions dominate the <em>Other</em> category, the model learns that these populations are structurally irrelevant.</p><h3><strong>Flattening Reality and the Geometry of Averaging</strong></h3><p>Machine learning systems reason in high-dimensional vector spaces. Categories correspond to regions; labels correspond to clusters. When distinct phenomena are collapsed into a single category, the algorithm computes an average representation — a centroid — that stands in for the group.</p><p>This centroid is a mathematical fiction.</p><p>When sadness and grief are merged, the resulting “average emotion” corresponds to no real human experience. It is neither transient nor enduring, neither sharp nor heavy. It is an abstraction that exists only because the preprocessing demanded it.</p><p>By forcing such averages, preprocessing erases boundaries and transitions. The model loses the ability to detect thresholds — the moment when sadness deepens into grief, or when grief begins to lift. The concept of <em>between-ness</em> disappears. What remains is a flattened emotional landscape optimized for computation, not understanding.</p><figure><img alt="Techniques for Visualizing High Dimensional Data – Serendipidata 🌱" src="https://cdn-images-1.medium.com/proxy/1*sNHaMTQBe3plUhk3k2dnYg.gif" /><figcaption>high dimensional data projection</figcaption></figure><h3><strong>Cultural Relativity and the Violence of Translation</strong></h3><p>Preprocessing often assumes that categories are universal. They are not.</p><p>Consider sentiment analysis applied across languages. The Portuguese word <em>saudade</em> is frequently translated as “sadness.” This translation is technically convenient and culturally incorrect. <em>Saudade</em> expresses nostalgic longing — an emotion that is simultaneously painful and warm, marked by absence but also by affection.</p><p>Forcing <em>saudade</em> into a negative sentiment category strips it of its duality. The resulting model learns that the emotion is purely negative. This is empirically false within the cultural context that produced the word, but true within the dataset constructed through preprocessing.</p><p>The harm here is not mistranslation; it is epistemological violence. A culturally specific reality is forced to conform to an alien taxonomy. The model becomes fluent in the dataset’s truth while remaining blind to the world it claims to represent.</p><h3><strong>Noise, Outliers, and the Epistemology of the “Normal”</strong></h3><p>Standard preprocessing pipelines often include outlier detection and removal. The justification is mathematical: extreme values distort estimates of central tendency. But this practice rests on a deeper assumption — that truth lies near the mean.</p><p>Statistical models implicitly privilege the center of the distribution. The tails are treated as errors, anomalies, or noise. Epistemologically, this encodes a utilitarian view of reality: what is true for the majority is the only truth worth modeling.</p><p>Socially, the tails often represent marginalized populations — people with rare diseases, atypical financial behaviors, or nonstandard living conditions. By trimming the tails to improve model accuracy, preprocessing erases these groups from the model’s reality.</p><p>A healthcare example makes this concrete. Heart attack datasets have historically been dominated by male patients. Their symptoms — chest pain, left arm numbness — become the canonical signal. Women, who often present with nausea, fatigue, or back pain, appear as outliers. If these cases are removed or downweighted to improve accuracy, the model constructs a world in which women’s heart attacks do not exist.</p><p>This is not a modeling failure. It is an epistemological choice disguised as optimization.</p><h3><strong>When Noise Is Resistance</strong></h3><p>Sometimes what appears as bad data is actually resistance to an ill-fitting system.</p><p>Imagine a social services form that requires a fixed address. Entries such as “No Fixed Abode,” “The Park,” or “Couch Surfing” violate the expected schema. A standard preprocessing response is to treat these as missing values and remove them.</p><p>The consequence is profound. The homeless population disappears from the analysis — not because they do not exist, but because their reality does not fit the database’s assumptions. The model learns only about those who already conform to institutional expectations.</p><p>In this way, preprocessing does not merely clean data. It enforces conformity.</p><h3><strong>Missing Data as Silence</strong></h3><p>Missing data is often treated as a nuisance — a pothole to be filled so algorithms can proceed. Techniques such as mean imputation or algorithmic estimation are deployed to “repair” the dataset.</p><p>Epistemologically, this is fiction-writing.</p><p>Filling a missing value with an average asserts that the missing individual is average. It creates a data point that never existed. In gentrifying neighborhoods, wealthy residents respond to surveys while displaced residents do not. If missing income values are imputed using the mean of visible data, poverty is mathematically erased. The dataset becomes populated with phantom middle-class residents.</p><p>Statisticians distinguish between data that is missing completely at random and data that is missing because of the value it would have had. Much socially significant data falls into the latter category. Substance abuse, domestic violence, undocumented labor, and mental illness are often missing precisely because they are stigmatized or dangerous to report.</p><p>Treating this silence as noise produces models that are systematically optimistic about suffering. The absence of data becomes evidence of absence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D8E3wmA3-UesZt2H.png" /><figcaption>missing data matrix visualization</figcaption></figure><h3><strong>Survivorship and the Truth of What Is Not Seen</strong></h3><p>One of the clearest demonstrations of epistemological silence is the World War II bomber analysis attributed to Abraham Wald. Engineers examined returning aircraft and proposed reinforcing areas with visible bullet holes. Wald observed that the missing planes — those that did not return — were the crucial data. The true vulnerability lay where there were no holes on surviving planes.</p><p>Modern data systems repeat this error constantly. Recruitment models trained on current employees learn the characteristics of those who survived hiring and retention processes. They ignore the missing data of capable candidates who were rejected or left due to hostile environments. The model reinforces the status quo by mistaking survival for merit.</p><h3><strong>The Myth of “Raw” Data</strong></h3><p>The term <em>raw data</em> suggests naturalness — data as something harvested from the world and later refined. This metaphor is false.</p><p>All data is already processed at the moment of collection. Decisions about what to measure, how to measure it, and which categories to use precede any dataset. Philosophers of science describe this as the theory-ladenness of observation: measurement is impossible without prior conceptual commitments.</p><p>In social data, this is stark. Crime datasets record street-level theft but rarely encode corporate malfeasance with comparable granularity. The resulting “raw” data teaches models that crime is something poor people do, not something powerful institutions do.</p><p>Sensors embed similar biases. Early color film was chemically optimized for white skin. Darker skin tones were underexposed not due to error, but due to design choices. Using such images as training data imports those biases into modern systems.</p><p>Data collection is not harvesting. It is photography. Every dataset is a snapshot taken from a particular angle, by a particular observer, for a particular purpose.</p><h3><strong>Epistemic Injustice and the Authority to Know</strong></h3><p>Epistemic injustice occurs when individuals are wronged in their capacity as knowers. In data-driven systems, this injustice is often automated.</p><p>Testimonial injustice arises when lived experience is discounted in favor of statistical inference. A patient reports chronic pain; an algorithm assigns a low probability of disability. The system treats the model’s output as objective truth and the person’s testimony as unreliable noise. Authority is transferred from the subject to the system.</p><p>Hermeneutical injustice occurs when experiences cannot be articulated because the categories to describe them do not exist. Before certain harms were named and measured, they were socially invisible. If data systems fail to collect information about police brutality, environmental poisoning, or workplace harassment, affected populations cannot prove their reality to the algorithmic gatekeepers of services and policy.</p><p>In both cases, data systems do not merely reflect injustice. They institutionalize it.</p><h3><strong>Implications for Practice</strong></h3><p>If preprocessing is epistemological, it demands a different standard of rigor:</p><ul><li>Preprocessing decisions should be documented as arguments, not procedures.</li><li>Sensitivity analyses should test how conclusions change under alternative pipelines.</li><li>Raw data and transformation histories should be preserved.</li><li>Domain experts and affected communities should inform category construction.</li><li>Missingness should be investigated, not automatically repaired.</li></ul><p>There is no neutral preprocessing — only more or less defensible preprocessing.</p><h3><strong>Conclusion</strong></h3><p>Data preprocessing is not janitorial work. It is architectural work. The individual who decides which categories collapse, which values are filled, and which cases are discarded writes the constitution of the model’s reality.</p><p>These decisions determine what differences matter, whose experiences are legible, and which truths the system can recognize. Treating preprocessing as epistemological work does not paralyze data science; it disciplines it. It replaces silent assumptions with explicit reasoning and optimization with accountability.</p><p>Good models begin not with cleaner data, but with more honest questions about what the data is allowed to mean.</p><h3>Further Reading &amp; Influences</h3><ul><li><em>Sorting Things Out — Geoffrey C. Bowker &amp; Susan Leigh Star</em><br> A foundational work on how classification systems encode social, political, and historical assumptions.</li><li><em>Epistemic Injustice — Miranda Fricker</em><br> Introduces testimonial and hermeneutical injustice, crucial for understanding how data systems silence certain knowers.</li><li><em>The Looping Effects of Human Kinds — Ian Hacking</em><br> Explores how categorization reshapes behavior and then feeds back into the category itself.</li><li><em>The Book of Why — Judea Pearl</em><br> Clarifies the limits of purely statistical reasoning and the role of assumptions in knowledge claims.</li><li><em>Statistical Rethinking — Richard McElreath</em><br> A clear articulation of how modeling choices encode beliefs about the data-generating process.</li><li><em>Weapons of Math Destruction — Cathy O’Neil</em><br> Examines how seemingly technical data decisions scale into structural harm.</li><li><em>A Method of Estimating Plane Vulnerability — Abraham Wald</em><br> A classic illustration of survivorship bias and the epistemology of missing data.</li></ul><blockquote><strong>Treating preprocessing as epistemological work shifts data science from optimization to accountability.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4740d50675b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Predictive Financial Advisor with a Hybrid AI Architecture]]></title>
            <link>https://medium.com/@goelanuj132.in/building-a-predictive-financial-advisor-with-a-hybrid-ai-architecture-53457df8b1d4?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/53457df8b1d4</guid>
            <category><![CDATA[personal-finance]]></category>
            <category><![CDATA[prophet]]></category>
            <category><![CDATA[isolation-forests]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[predictive-modeling]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Fri, 15 Aug 2025 19:00:56 GMT</pubDate>
            <atom:updated>2025-08-15T19:00:56.214Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From a 50,000-row spreadsheet to a live web app that predicts your financial future, finds hidden spending risks, and gives you personalized advice with </em>Prophet, Scikit-learn, and <em>Gemini.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0vPf2MYmtLwM_uScIsgCLw.png" /></figure><p>We’ve all been there. You open your banking app, scroll through an endless list of cryptic transactions, and try to answer a seemingly simple question: “Am I doing okay financially?”</p><p>The data is all there, but it’s just… data. It’s a record of the past, not a guide for the future. I found myself wishing for a tool that could do more than just show me a list of my expenses. What if an app could look at my messy transaction history, understand my habits, and tell me where I was headed? What if it could act like a real financial advisor?</p><p>Fueled by this question and a tight deadline for a prototype challenge, I spent four days building exactly that: a Personalized AI Finance Advisor. This is the story of how I went from a 50,000-row bank statement to a fully deployed web application that uses machine learning and generative AI to provide financial clarity.</p><h3>The Blueprint: Designing an App That’s Both Smart and Fast</h3><p>The biggest challenge with any data-intensive application is performance. A user won’t wait five minutes for an analysis, no matter how brilliant it is. This led me to a crucial architectural decision: a <strong>hybrid architecture</strong>.</p><p>Think of it like a restaurant. You have the dining area (the frontend), where the customer experience needs to be fast and smooth, and the kitchen (the backend), where all the heavy, time-consuming work happens.</p><ul><li><strong>The Frontend (The Dining Area):</strong> A lightweight and interactive user interface built with <strong>Streamlit</strong>. Its only job is to take the user’s order (the uploaded file) and present the final meal (the analysis).</li><li><strong>The Backend (The Kitchen):</strong> A powerful and scalable service built with <strong>FastAPI</strong>. This is where all the magic happens: data cleaning, running machine learning models, and calling the generative AI.</li></ul><p>This separation ensures the user interface never gets bogged down, even when the backend is analyzing years of transaction data.</p><h3>The Tech Deep Dive: Tools of the Trade</h3><p>For those interested in the technical details, here’s a look under the hood.</p><h3>The Backend: The Brains of the Operation</h3><p>The entire backend was built in Python and deployed as a containerized application on <strong>Google Cloud Run</strong>.</p><ul><li><strong>FastAPI:</strong> I chose FastAPI as the web framework because it’s incredibly fast, modern, and comes with automatic documentation, which is perfect for rapid prototyping.</li><li><strong>Pandas:</strong> For the most critical and often underestimated part of any data project: <strong>data cleaning</strong>. The backend had to be robust enough to handle messy, real-world bank statements — files with different encodings, missing values, and oddly formatted numbers. Pandas was the perfect tool for this.</li><li><strong>Prophet &amp; Scikit-learn:</strong> These are the workhorses of the machine learning pipeline. <strong>Prophet</strong> is a powerful time-series forecasting library developed by Facebook, which I used to predict future cash flow. <strong>Scikit-learn</strong> is the gold standard for classical machine learning, and I used its IsolationForest model for anomaly detection.</li><li><strong>Google’s Gemini:</strong> This is the generative AI that powers the final report. I chose the Pro model for its advanced reasoning and ability to generate high-quality, structured text.</li></ul><h3>The Frontend: The User’s Cockpit</h3><p>The frontend is built entirely in Python using <strong>Streamlit</strong>. This allowed for incredibly fast development, turning data and models into an interactive web app without writing a single line of JavaScript.</p><ul><li><strong>Streamlit:</strong> Perfect for creating data-centric applications. It handles all the complexity of building a web UI, allowing me to focus on the logic.</li><li><strong>Plotly:</strong> For all the data visualizations. Plotly creates beautiful, fully interactive charts (like the forecast cone and the spending pie chart) that make the data easy to understand.</li></ul><h3>The “Magic”: How the AI Actually Works</h3><p>The application’s intelligence comes from a three-stage AI/ML pipeline that runs on the backend.</p><h3>1. The Fortune Teller: Prophet for Forecasting</h3><p>The first step is to predict the future. The Prophet model is brilliant at this because it’s designed to understand human-scale time-series data. It automatically detects weekly, monthly, and yearly patterns in the transaction history.</p><p>To make this fast, I used a technique called <strong>resampling</strong>. Instead of feeding the model all 50,000+ individual transactions, I first grouped them by day to get the net daily cash flow. This reduced the dataset size by over 95%, cutting the analysis time from minutes to seconds while actually improving the model’s ability to see the high-level trends.</p><h3>2. The Detective: IsolationForest for Anomaly Detection</h3><p>Next, I needed to find the “financial leaks” — the unusual transactions that could represent risks or opportunities for savings. The IsolationForest model is perfect for this.</p><p>Its core idea is simple: <strong>anomalies are easier to separate from the crowd.</strong> The model builds hundreds of random decision trees, and the transactions that are “isolated” in the fewest steps are flagged as the most unusual. After a full analysis, I programmed the backend to return only the <strong>top 10 most severe anomalies</strong>, turning a mountain of data into a focused, actionable list.</p><h3>3. The Strategist: Gemini for the Report</h3><p>This is where the project truly becomes an “AI Advisor.” Raw data and charts are useful, but they don’t tell you what to <em>do</em>. To solve this, I engineered a <strong>multi-prompt chain</strong> for the Gemini API.</p><p>Instead of one generic prompt, I make three separate, specialized API calls:</p><ul><li><strong>The Analyst:</strong> The first prompt asks Gemini to act as a data analyst, explaining the key financial metrics and the forecast trend in simple, educational terms.</li><li><strong>The Investigator:</strong> The second prompt asks it to act as a forensic accountant, providing a plausible explanation for why each of the top 10 anomalies was flagged.</li><li><strong>The Strategist:</strong> The final prompt asks it to act as a financial strategist, taking all the previous context and generating at least three detailed, non-generic recommendations, each with clear reasoning, pros, and cons.</li></ul><p>This technique results in a final report that is incredibly detailed, structured, and genuinely useful.</p><h3>The “Wow” Moment: Interactive Scenario Planning</h3><p>The feature I’m most proud of is the <strong>“What-If” Scenario Planner</strong>. The dashboard includes interactive sliders for the user’s top spending categories. As the user moves a slider — for example, to simulate a 20% reduction in “Food &amp; Dining” expenses — the forecast chart updates <strong>in real-time</strong> to show them exactly how that decision will impact their bank balance 90 days from now.</p><p>This transforms the app from a passive report into an active, engaging financial planning tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hqQSGVP_1c_m5f6U8p3q3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2vyq4t-mHIyPBkMpBNb9w.png" /><figcaption>Peek into Financial dashboard of this app</figcaption></figure><h3>The Takeaway</h3><p>This four-day sprint was an incredible journey. It reinforced my belief that the future of AI isn’t just about building powerful models, but about thoughtfully combining different AI techniques — predictive, analytical, and generative — to create complete, end-to-end solutions that solve real-world problems.</p><p>Thank you for reading! Feedback and reviews are always welcomed.</p><p><em>You can try out the live application here: <br></em><a href="https://personal-finance-advisor-ai.streamlit.app/">https://personal-finance-advisor-ai.streamlit.app/</a></p><p><em>The full source code is available on my GitHub: </em><a href="https://github.com/goelanuj371/ai-finance-advisor">https://github.com/goelanuj371/ai-finance-advisor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53457df8b1d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your Own Customizable RAG Pipeline in Python — With Just a YAML File]]></title>
            <link>https://medium.com/@goelanuj132.in/build-your-own-customizable-rag-pipeline-in-python-with-just-a-yaml-file-71af4e06b2e1?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/71af4e06b2e1</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[yaml]]></category>
            <category><![CDATA[python-libraries]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 19:41:13 GMT</pubDate>
            <atom:updated>2025-06-30T19:41:13.701Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build Your Own Customizable RAG Pipeline in Python — With Just a YAML File</h3><p>A plug-and-play Python library for building Retrieval-Augmented Generation (RAG) pipelines with just a YAML file — seamlessly integrates OpenAI, Gemini, HuggingFace, and more for fast, flexible, and multi-provider document-based Q&amp;A.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LM2YTv4uHR7hR-q_CJajQ.png" /></figure><p>Retrieval-Augmented Generation (RAG) is a powerful approach to enhance language models with external knowledge, especially from user-provided documents. However, most RAG pipelines are locked into a specific LLM or require tedious rewrites to support different providers.</p><p>What if you could install a Python library, configure it with a YAML file, and instantly switch between OpenAI, Gemini, HuggingFace, xAI, or local models?</p><p>Introducing customrag — a highly flexible and developer-friendly RAG pipeline in Python designed for easy configuration and fast setup.</p><h3>What Is customrag?</h3><p>customrag is a pip-installable Python package that allows you to construct and query RAG pipelines with minimal boilerplate. It features:</p><ul><li>Support for multiple LLMs and embedding models (OpenAI, Gemini, HuggingFace, xAI, and local sentence-transformers)</li><li>Ingestion of various document types (.pdf, .docx, .txt, .json, .csv, .md)</li><li>A simplified Python API built on top of LangChain</li><li>Native Gemini Cloud Console SDK support</li><li>Configuration managed entirely via a YAML file</li></ul><h3>Why I Built It</h3><p>While working on LLM-based projects, I encountered a recurring challenge — RAG pipelines that were hardcoded, inflexible, or incompatible with the Gemini Cloud SDK. Switching between APIs often meant rewriting large sections of code.</p><p>customrag was created to address these issues through:</p><ul><li>Modularity — plug-and-play support for different models</li><li>Simplicity — a one-command setup and YAML-driven configuration</li><li>Compatibility — support for both LangChain and native SDKs</li></ul><h3>Getting Started</h3><h3>1. Install the package</h3><pre>pip install customrag</pre><h3>2. Generate a starter config</h3><pre>customrag-setup</pre><p>This will create a config.yaml in your current directory.</p><h3>3. Configure your providers and API keys</h3><pre>embedding:<br>  provider: gemini<br>  model: models/embedding-001</pre><pre>llm:<br>  provider: gemini<br>  model: gemini-1.5-pro</pre><pre>api_keys:<br>  gemini: your_gemini_cloud_key<br>  openai: your_openai_key<br>  huggingface: your_hf_token<br>  xai: your_xai_key</pre><p>Switching providers only requires a change in this file — no need to modify your code.</p><h3>4. Use the pipeline in your script</h3><pre>from customrag import RAGPipeline</pre><pre>pipeline = RAGPipeline(&quot;config.yaml&quot;)<br>pipeline.build_vectorstore(&quot;resume.pdf&quot;)<br>response = pipeline.query(&quot;What are my key skills?&quot;)<br>print(response)</pre><h3>Key Features</h3><ul><li>Seamless provider switching: Gemini, OpenAI, HuggingFace, xAI</li><li>Multi-format document ingestion</li><li>FAISS-based vector indexing</li><li>YAML config-driven setup</li><li>Native Gemini SDK fallback when LangChain doesn’t support it</li></ul><h3>Gemini SDK Support</h3><p>Most LangChain tools don’t support the Gemini Cloud Console SDK — they rely on Gemini Studio. customrag detects if the SDK should be used and automatically switches to native API calls like this:</p><pre>client = genai.Client(api_key=your_key)<br>response = client.models.generate_content(model=&quot;gemini-1.5-pro&quot;, contents=prompt)</pre><p>This logic is handled internally, so your code remains clean and consistent.</p><h3>When Should You Use customrag?</h3><ul><li>When building document-based Q&amp;A systems</li><li>When prototyping with different LLMs and embeddings</li><li>When you need quick setup without sacrificing flexibility</li><li>When your team wants an extendable RAG tool with modern APIs</li></ul><h3>Under the Hood</h3><p>The project includes:</p><ul><li>rag_pipeline.py: Core RAG logic</li><li>customrag_setup.py: CLI tool to generate YAML config</li><li>__init__.py: Package init file</li><li>setup.py: Python packaging and distribution logic</li></ul><h3><strong>Architecture Diagram</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XPVmxbgQ4Ic1C9UicAqifA.png" /><figcaption>Architecture Diagram</figcaption></figure><h3>Future Plans</h3><p>Upcoming improvements may include:</p><ul><li>Streaming response support</li><li>Additional vectorstore backends (e.g., Chroma, Weaviate)</li><li>A basic web UI using Streamlit or Gradio</li><li>Better logging and error handling</li></ul><h3>Feedback and Contributions</h3><p>I’ve tested this mainly with Gemini Cloud Console APIs. If you use it with OpenAI, HuggingFace, xAI, or Gemini Studio, I would love to hear your experience.</p><p>Contributions are welcome! Visit the GitHub repo:</p><p><strong>GitHub</strong>: <a href="https://github.com/goelanuj371/customrag">https://github.com/goelanuj371/customrag</a></p><h3>Links</h3><ul><li>PyPI: <a href="https://pypi.org/project/customrag/">https://pypi.org/project/customrag/</a></li><li>GitHub: <a href="https://github.com/goelanuj371/customrag">https://github.com/goelanuj371/customrag</a></li></ul><p>If you’re exploring RAG pipelines or working with LLMs across providers, give customrag a try. It&#39;s built for developers like you who want performance, portability, and simplicity — without sacrificing power.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71af4e06b2e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing SpaceLab AI: A Smart Assistant That Translates Space Tech into Real-World Innovation]]></title>
            <link>https://medium.com/@goelanuj132.in/introducing-spacelab-ai-a-smart-assistant-that-translates-space-tech-into-real-world-innovation-8517c20ffb7d?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/8517c20ffb7d</guid>
            <category><![CDATA[nasa]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[space]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Fri, 27 Jun 2025 10:56:28 GMT</pubDate>
            <atom:updated>2025-06-27T15:22:59.003Z</atom:updated>
            <content:encoded><![CDATA[<p>A Lightweight RAG Architecture Using Gemini &amp; FAISS for NASA Tech Discovery</p><h3>A Peek into the Interface</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h6vMZtyvyR4OgpHY25YMGg.png" /><figcaption>Home Screen</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NI2usvO6MAOZvC0oG_ZSRg.png" /><figcaption>Working Interface</figcaption></figure><p>For the past few days, I’ve been working late into the night, driven by an idea:<br><strong>What if we could take the cutting-edge R&amp;D done at NASA and turn it into accessible knowledge for innovators, startups, and curious builders?</strong></p><p>That idea became <strong>SpaceLab AI</strong> — a smart, searchable, conversational assistant built using Retrieval-Augmented Generation (RAG), vector databases, and Google’s Gemini models.</p><h3>What is SpaceLab AI?</h3><p><strong>SpaceLab AI</strong> is not your typical chatbot.</p><p>While large models like ChatGPT or Gemini are general-purpose and trained on broad internet data, SpaceLab AI is <strong>focused, grounded, and domain-specific</strong>.</p><p>It’s an <strong>innovation assistant</strong> trained to understand and answer questions based on:</p><ul><li><strong>NASA TechPort</strong> — NASA’s internal database of funded R&amp;D projects</li><li><strong>NASA TechTransfer</strong> — Publicly available patents, licenses, and commercial spin-offs</li></ul><p>Think of it as a <strong>chat interface for NASA’s innovation database</strong> — a way to talk directly to the research happening in space tech.</p><h3>Why does it matter?</h3><p>Most AI chatbots today are great at general knowledge, but <strong>when it comes to precision, domain expertise, and citations, they fall short</strong>.</p><p>Let’s say you’re working on:</p><ul><li><strong>Affordable clean energy</strong> and want to explore how solar panel designs from satellites could help</li><li><strong>Robotics for disaster zones</strong>, and you’re curious about autonomous drones used on Mars</li><li><strong>Material science</strong>, and you’re wondering what radiation-resistant materials are used in space</li></ul><p>SpaceLab AI lets you ask these questions and <strong>get answers based on actual NASA research</strong>, not just internet approximations.</p><h3>Key Features</h3><h3>1. Vector Search over NASA Data</h3><p>Behind the scenes, SpaceLab AI indexes over 17,000+ records from TechPort and TechTransfer using <strong>FAISS</strong> and <strong>Google Generative AI Embeddings</strong>.</p><p>This enables semantically rich search — meaning it doesn’t just match keywords, but understands <em>meaning</em>.</p><h3>2. Dual Source Retrieval</h3><p>Each query is run against two separate knowledge bases:</p><ul><li><strong>TechPort (R&amp;D Projects)</strong></li><li><strong>TechTransfer (Patents &amp; Applications)</strong></li></ul><p>Both sources are searched in parallel, and the context is intelligently fused before the model responds.</p><h3>3. Gemini-powered RAG Assistant</h3><p>The model used is <strong>Gemini </strong>— chosen over faster versions for <strong>higher accuracy and better factual grounding</strong>.</p><p>Instead of asking the model to hallucinate an answer, we follow the <strong>RAG (Retrieval-Augmented Generation)</strong> technique:</p><ul><li>Retrieve the most relevant documents</li><li>Format them into a system-aware prompt</li><li>Let Gemini generate grounded answers</li></ul><h3>4. Session Memory</h3><p>Every query and response is stored in <strong>Streamlit session state</strong>, providing lightweight memory and a visible chat history.</p><p>This makes the experience feel more <strong>conversational and interactive</strong>.</p><h3>5. Clean, Modern UI with Streamlit</h3><p>Built with <strong>Streamlit</strong>, the UI is:</p><ul><li>Minimalistic</li><li>Intuitive</li><li>Web-deployable with just one command</li></ul><p>You can deploy your own version and customize the assistant for your industry or research area.</p><h3>Architecture Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZhuDhIj4qgsL8SdLtIgUg.png" /><figcaption>Architecture Diagram</figcaption></figure><h3>How I Built It — A Technical Overview</h3><p>Building <strong>SpaceLab AI</strong> was all about bridging two worlds:<br> → The <strong>vast, underutilized domain knowledge</strong> from NASA<br> → And the <strong>modern capabilities of retrieval-augmented AI systems</strong></p><p>Here’s how I approached it step by step:</p><h3>1. Preparing the NASA Knowledge Base</h3><p>I began with two datasets:</p><ul><li><strong>TechPort</strong>: A massive dataset containing NASA’s internal research and development projects over the years.</li><li><strong>TechTransfer</strong>: A set of publicly available patents and spinoffs developed from NASA technologies.</li></ul><p>Since these datasets are not designed for conversational AI, I had to <strong>clean, parse, and convert them</strong> into structured chunks of text. Each chunk had to be small enough for a language model to process, but rich enough to convey context.</p><p>To enable semantic search (where the assistant understands meaning, not just keywords), I embedded this content into vector representations. This allowed the assistant to retrieve relevant data even if the user phrased their question differently from the original NASA documentation.</p><h3>2. Creating a Searchable Vector Database</h3><p>With embedded vectors in hand, I created two separate <strong>search indices</strong>:</p><ul><li>One for TechPort</li><li>One for TechTransfer</li></ul><p>These indices act like a memory bank — so when a user asks a question, the assistant doesn’t search blindly. It runs a <strong>semantic search</strong> through this vector database to find the most relevant research and patent data for that question.</p><p>This is what powers the system’s factual accuracy and domain specificity.</p><h3>3. Crafting the Retrieval-Augmented Answer Engine</h3><p>Once I had a working search pipeline, I focused on the <strong>answer generation system</strong>. Here’s what happens behind the scenes:</p><ul><li>It then <strong>constructs a custom prompt</strong> — a kind of instruction + evidence bundle — that is passed to <strong>Google Gemini Pro</strong>, a state-of-the-art large language model.</li><li>This model reads the documents like a researcher and generates an answer that is <strong>grounded in the retrieved sources</strong>, not guessed or hallucinated.</li><li>The result? Answers that feel natural, like talking to a human assistant, but are always based on <strong>real NASA projects or patents</strong>.</li></ul><h3>4. Making It Conversational and Clean</h3><p>I didn’t just want this to be a search engine. I wanted it to feel like an intelligent assistant.</p><p>So I implemented a lightweight memory system that stores the ongoing conversation. Each question and answer pair is recorded in the session, allowing for follow-up questions and context awareness (within a session).</p><p>Finally, I used <strong>Streamlit</strong> to create a clean, interactive web app that feels like a chat — but underneath, it’s powered by retrieval, ranking, and generation workflows.</p><h3>Why It’s Unique</h3><p>Most AI apps built today are general-purpose.</p><p><strong>SpaceLab AI is different:</strong></p><ul><li><strong>Niche use case</strong>: Focused entirely on turning government-funded space tech into usable knowledge</li><li><strong>Cited sources</strong>: Shows actual R&amp;D projects and patents in the response</li><li><strong>Deployable &amp; Modular</strong>: Anyone can fork the repo and build their own assistant with different data</li><li><strong>No hallucinations</strong>: Every answer is grounded in vector-searched documents</li></ul><h3>Try It Out</h3><p>🔗 <strong>Live Demo</strong>: <a href="https://spacelab-ai.streamlit.app/">https://spacelab-ai.streamlit.app/</a></p><h3>Final Thoughts</h3><p>I built <strong>SpaceLab AI</strong> in under 6 hours from idea to deployment — with late nights, debugging loops, and a lot of coding. But more than the code, this project showed me:</p><blockquote><em>A focused AI assistant trained on high-quality data is more powerful than a generic chatbot trained on everything.</em></blockquote><p>If you’re building in clean tech, aerospace, robotics, or even education — there’s a lot NASA has done that’s just waiting to be discovered.</p><p>SpaceLab AI is my small attempt to bring that to light.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8517c20ffb7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building “Gita Life Guide AI Chatbot”: How I Used LLMs to Turn Bhagavad Gita into a Personal…]]></title>
            <link>https://medium.com/@goelanuj132.in/building-gita-life-guide-ai-chatbot-how-i-used-llms-to-turn-bhagavad-gita-into-a-personal-cee6a883553e?source=rss-4f4f4b9f9cbe------2</link>
            <guid isPermaLink="false">https://medium.com/p/cee6a883553e</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[bhagavad-gita]]></category>
            <category><![CDATA[ai-chatbot]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Anuj Goel]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 18:21:51 GMT</pubDate>
            <atom:updated>2025-06-25T18:21:51.159Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building “Gita Life Guide AI Chatbot”: How I Used LLMs to Turn Bhagavad Gita into a Personal Reflection Assistant</strong></h3><p>In a time where AI is often associated with productivity tools, chat assistants, or entertainment bots, I wanted to build something more personal, meaningful, and rooted in timeless wisdom. That’s how <strong>Gita Life Guide</strong> was born — an AI-powered web application that reflects on your personal life challenges using the teachings of the <strong>Bhagavad Gita</strong>.</p><h3>The Vision</h3><p>We all go through moments of doubt, frustration, or confusion. While ancient texts like the Gita hold immense wisdom, most people don’t know how to relate its verses to everyday struggles. My goal was to bridge this gap by combining <strong>modern language models</strong> with <strong>classical Indian philosophy</strong>, creating a space for thoughtful, personalized reflection.</p><h3>What the App Does</h3><p><strong>Gita Life Guide</strong> lets a user enter any thought or life problem they’re facing. The app then:</p><ol><li><strong>Semantically searches</strong> for the most relevant Bhagavad Gita verse.</li><li>Displays the Sanskrit verse and its English translation.</li><li>Uses <strong>Google’s Gemini 2.5 model</strong> to generate a compassionate, context-aware reflection inspired by the verse.</li></ol><p>This means you’re not just chatting with an AI — you’re receiving reflections rooted in thousands of years of wisdom.</p><h3>Tech Stack Breakdown</h3><ul><li><strong>Frontend &amp; Deployment</strong>: Built with <strong>Streamlit</strong>, making it responsive across laptops and mobiles</li><li><strong>LLM Integration</strong>: Used <strong>Gemini 2.5 Flash</strong> via <strong>Google Generative AI SDK</strong></li><li><strong>Vector Search</strong>: Indexed verse embeddings using <strong>FAISS</strong> for similarity search</li><li><strong>Embeddings</strong>: Generated using GoogleGenerativeAIEmbeddings (models/embedding-001)</li><li><strong>Orchestration</strong>: Powered by langchain to build the reasoning logic</li><li><strong>Version Control</strong>: Managed with Git and GitHub</li></ul><h3>Features Implemented in v1:</h3><ul><li>Takes real-life problems as input and responds with relevant Gita verses</li><li>Shows both Sanskrit and English translation</li><li>AI-powered compassionate response based on context</li></ul><h3>Challenges Faced</h3><ul><li><strong>Dependency Management</strong>: Streamlit deployment was initially tricky due to version conflicts in the requirements.txt. Careful pruning of dependencies and version alignment fixed this.</li><li><strong>Gemini API Usage</strong>: Switching from older models like chat-bison-001 to newer ones like gemini-2.5-flash required digging into Google&#39;s official documentation and adapting the code.</li><li><strong>Vector Search Tuning</strong>: I had to balance performance and relevance when embedding 700+ Gita verses.</li></ul><h3>What Makes This Unique?</h3><p>You might wonder, “Why not just use ChatGPT or Gemini directly?” Here’s what sets this apart:</p><ul><li><strong>Personalization via Verse Matching</strong>: Instead of generic advice, each answer is grounded in a <strong>specific verse</strong> of the Gita.</li><li><strong>Custom Embedding Search</strong>: Created a full FAISS-based semantic search system trained specifically on Gita verses.</li><li><strong>Contextual Generation</strong>: Reflections are not just LLM outputs — they are tightly guided by spiritual context through custom prompts.</li><li><strong>No External UI Toolkits</strong>: Fully built and deployed via <strong>Streamlit</strong> without needing HTML/CSS or JS knowledge.</li></ul><h3>Roadmap Ahead</h3><p>The current version is only the beginning. I’m actively working on:</p><ul><li><strong>Multi-verse Retrieval</strong>: To offer layered insights from multiple verses</li><li><strong>Chat History Memor</strong>y: Past messages will influence future reflections</li><li><strong>Multi-query Conversations</strong>: Allowing deeper spiritual exploration across messages</li><li><strong>Personalization Layer</strong>: Adapting responses based on user’s emotional tone and query patterns</li></ul><h3>Lessons Learned</h3><p>This project gave me real-world experience in:</p><ul><li>Prompt engineering for emotionally grounded language</li><li>Handling dependency hell and environment conflicts</li><li>Working with modern vector search architecture</li><li>Thinking about AI alignment with human emotions and values</li></ul><h3>Try It Yourself</h3><ul><li><strong>🔗</strong><a href="https://geeta-life-guide-ai-chat-awkfvqhymwnzkusbfkdabo.streamlit.app/"><strong> Live App</strong></a></li></ul><h3>Final Thoughts</h3><p>“Gita Life Guide” is not a religious tool. It’s a <strong>reflection assistant</strong> — one that helps users navigate their problems with peace, purpose, and ancient wisdom. Whether you are a developer, a student, or just someone looking for clarity, I hope this project inspires you to explore how AI can serve not just productivity, but also <strong>human introspection</strong>.</p><p>Thanks for reading ✨. Would love your feedback or collaboration ideas!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cee6a883553e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>