<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xwang on Medium]]></title>
        <description><![CDATA[Stories by Xwang on Medium]]></description>
        <link>https://medium.com/@xwang222?source=rss-efca2a3557f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LLaVtG27yWpAyPu_FuffQQ.png</url>
            <title>Stories by Xwang on Medium</title>
            <link>https://medium.com/@xwang222?source=rss-efca2a3557f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:05:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xwang222/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DS Snapshot: Making Data Science Work Easier to Archive and Review]]></title>
            <link>https://medium.com/@xwang222/ds-snapshot-making-data-science-work-easier-to-archive-and-review-29c47f56c800?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/29c47f56c800</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[vscode-extension]]></category>
            <category><![CDATA[productivity-tools]]></category>
            <category><![CDATA[evidence-archive]]></category>
            <category><![CDATA[jupyterlab]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 03:47:50 GMT</pubDate>
            <atom:updated>2026-07-17T03:50:33.057Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently built <strong>DS Snapshot</strong>, a tool for archiving and managing evidence from analytical and technical work.</p><p>DS Snapshot currently runs inside <strong>JupyterLab 4.0+</strong> and <strong>VS Code</strong> as the capture interface. The evidence itself does not have to come from Python or notebook execution. <strong>You can use the notebook or editor simply as an entry point</strong> to organize supporting materials such as spreadsheets, SQL files, screenshots, charts, logs, notes, and generated outputs.</p><p>The idea is simple: data science work should be easier to review, explain, and revisit later. DS Snapshot helps capture the notebooks, scripts, outputs, screenshots, notes, and other supporting materials around a project, and organizes them into a structured snapshot.</p><h3>Why I Built It</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOIFVUcu0hSDajn5iBRrHg.png" /></figure><p>Data science work is often difficult to reconstruct after the fact.</p><p>The final deliverable may be a clean notebook, a report, or a model output, but the actual work usually happens across many places. There may be exploratory notebooks, saved charts, screenshots, intermediate outputs, markdown notes, logs, and small decisions made along the way.</p><p>When someone asks later, “How did we get this result?” or “What changed from the previous version?” the evidence is often scattered across different files and folders.</p><p>Git is helpful for tracking code changes, but data science evidence is not only code. A chart, a notebook output, a screenshot, or a short note may be just as important for explaining the work.</p><p>That is the problem DS Snapshot is trying to solve.</p><h3>What DS Snapshot Does</h3><p>DS Snapshot creates a structured archive of project evidence.</p><p>It can help collect and organize materials such as notebooks, scripts, markdown notes, generated outputs, screenshots, and other supporting project artifacts.</p><p>The snapshot is created separately from the original project, so it does not overwrite the source files.</p><p>DS Snapshot is not meant to replace JupyterLab, VS Code, Git, MLflow, DVC, or documentation tools. It is designed to complement them by making the evidence trail around data science work easier to manage.</p><h3>Demo</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F78vUEQvzyrg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D78vUEQvzyrg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F78vUEQvzyrg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a46d66b9973954224f00523b6232a4c7/href">https://medium.com/media/a46d66b9973954224f00523b6232a4c7/href</a></iframe><p>Example use cases:</p><ol><li>Panel overview: <a href="https://youtu.be/bPMHgIceMok">here</a></li></ol><p>2. Workflow with Python: <a href="https://youtu.be/SXMhVT5giOI">here</a></p><p>3. Workflow without Python: <a href="https://youtu.be/hEOJQ7qF-t8">here</a></p><p>4. Audit AI work: <a href="https://youtu.be/H1J1FrBiwo0">here</a></p><p>5. Model Comparison: <a href="https://youtu.be/kNCk9yL-Ae8">here</a></p><p>6. Evidence Management: <a href="https://youtu.be/HPBtkpNvNWg">here</a></p><h3>Installation</h3><ul><li><strong>JupyterLab 4.0+</strong></li></ul><p>pip install jupyterlab-ds-snapshot</p><ul><li><strong>VS Code</strong></li></ul><p>Search for <strong>DS Snapshot</strong> in the Extensions view and install it directly</p><h3>Ending Remarks</h3><p>If you work on data science projects and often need to review, explain, or organize your work evidence later, DS Snapshot may be useful for your workflow. Give it a try, and see whether it helps when you revisit a project a few weeks later and need to remember what you did, what the results were, and why you made certain decisions. Please leave your comments below and tell me if you like it or not and how to further improve. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29c47f56c800" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Built a Local Context Generator to Help AI Agents Save Tokens]]></title>
            <link>https://medium.com/@xwang222/i-built-a-local-context-generator-to-help-ai-agents-save-tokens-f04bc5bf5da7?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f04bc5bf5da7</guid>
            <category><![CDATA[token-saving]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[obsidian]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Wed, 24 Jun 2026 02:59:19 GMT</pubDate>
            <atom:updated>2026-06-24T02:59:19.357Z</atom:updated>
            <content:encoded><![CDATA[<p>AI agents are powerful, but their context is not free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KajKqvgHieWpWhAWg1Mbag.png" /></figure><p>When I use agents on local projects or notes, I often see them start with basic discovery: listing directories, running glob searches, checking file names, and trying to understand the workspace structure. This step is necessary, but it also feels wasteful.</p><p>Directory discovery is mostly local metadata work. It does not require deep reasoning, an LLM call, or a high-quality semantic summary. Yet for a large vault or project folder, the agent may spend meaningful context just figuring out the basic map.</p><p>That made me wonder: <strong>why not do this cheap discovery locally first?</strong></p><h3>The idea</h3><p>Agents are powerful, but not every step in an agent workflow needs an LLM. Some tasks are simple for humans, cheap for local code, and expensive only because we ask the agent to do them inside its context window. Directory discovery is one of those tasks.</p><p>Before an agent can reason over a project or a knowledge base, it often needs a basic map: what exists, where things are, and which areas look worth reading. That map does not have to be perfect. It does not need semantic understanding. It just needs to be good enough to help the agent choose where to look next.</p><p>So I built a plugin for Obsidian (<a href="https://github.com/xwang0176/obsidian-agent-context">Link</a>) that turns this discovery step into a local pre-step. A standalone Python version is available <a href="https://github.com/xwang0176/agent-context-indexer">here</a>.</p><p>Instead of asking the agent to explore the vault from scratch, the plugin generates a local context index first. The agent can read that index, decide whether it helps, and then choose a smaller set of notes or files to open.</p><h3>What my plugin does</h3><p>The Obsidian plugin is called <strong>Obsidian Agent Context (</strong><a href="https://github.com/xwang0176/obsidian-agent-context"><strong>Link</strong></a><strong>)</strong>. The plugin does NOT call an LLM, does NOT generate embeddings, does NOT upload notes and does NOT overwrite existing AGENTS.MD.</p><p>It runs <strong>locally </strong>inside Obsidian and turns the vault into an agent-readable context index. Instead of asking the agent to explore the vault from scratch, the plugin gives it a structured starting point.</p><p>The plugin can generate context at three levels:</p><ol><li>a broad index for the whole vault</li><li>focused indexes for one or more configured folders</li><li>both the vault-level index and the configured folder-level indexes together</li></ol><p>All generated files live under .agent_context/. Together, they form a local map of the vault and selected folders, helping the agent understand where things are, what looks relevant, and which notes or files are worth opening next.</p><p>The intended workflow is as simple as follows:</p><pre>1. Generate a local index<br>2. Ask the agent whether the index is useful<br>3. If useful, keep or merge the generated AGENTS pointer</pre><p>A typical prompt after scanning might be:</p><pre>Please read files in .agent_context/latest/ first.<br>Does this index help you <br>- understand the vault structure, <br>- choose which notes to open next, <br>- and avoid unnecessary token usage?</pre><p>This turns the index into something testable. If the agent says it is useful, keep it. If not, delete it. You can do it manually, or ask your agent to do that for you.</p><pre>Please merge the generated AGENTS pointer with the existing AGENTS.MD</pre><h3>A quick demo</h3><p>I made a short two-minute demo to show the workflow end to end.</p><p>The video covers three things: manually installing the plugin from a GitHub release, generating a local Agent Context index inside Obsidian, and asking an agent to evaluate whether the generated index helps it understand the vault structure and choose what to open next.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5mtxGRhqxA4&amp;type=text%2Fhtml&amp;schema=google&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5mtxGRhqxA4" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/77b3fbb1b574e6bb087e902e8fb1c02e/href">https://medium.com/media/77b3fbb1b574e6bb087e902e8fb1c02e/href</a></iframe><h3>I also built a Python version</h3><p>Along with the Obsidian plugin, I built a standalone Python version called <strong>Agent Context Indexer (</strong><a href="https://github.com/xwang0176/agent-context-indexer"><strong>Link</strong></a><strong>)</strong>.</p><p>The Python version is for general local folders: code repositories, research folders, data projects, and mixed document directories.</p><p>The Obsidian version is more vault-native. It is designed around Obsidian notes, wikilinks, frontmatter, folder-level scans, and command-palette usage.</p><p>They share the same core idea:</p><blockquote><em>Before an agent reads deeply, give it a local, low-cost, navigable context map.</em></blockquote><h3>Ending remarks</h3><p>Agents are very good at reasoning, editing, and connecting information across files. But not every part of the workflow needs to happen inside the model context. Some steps are better handled locally first.</p><p>Obsidian Agent Context (<a href="https://github.com/xwang0176/obsidian-agent-context">Link</a>) is my attempt to move that cheap discovery work out of the context window and into a local, inspectable index. The goal is not to replace agent reasoning. The goal is to give the agent a better starting point before it starts reading.</p><p>This is an early public beta. If you have any comments or suggestions, I would love to hear them. If you find this tool useful, please feel free to share it with others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f04bc5bf5da7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why More Control Can Hurt: Understanding Bad Controls Through Bayesian Networks]]></title>
            <link>https://medium.com/@xwang222/why-more-control-can-hurt-understanding-bad-controls-through-bayesian-networks-9feb3af6ab54?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9feb3af6ab54</guid>
            <category><![CDATA[directed-acyclic-graph]]></category>
            <category><![CDATA[econometrics]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Sat, 03 May 2025 15:58:19 GMT</pubDate>
            <atom:updated>2025-05-03T15:58:19.432Z</atom:updated>
            <content:encoded><![CDATA[<p>Whether evaluating a new product or a regulatory change, getting the causal structure right is essential. To satisfy the conditional independence assumption in causal inference, controlling for covariates is often necessary. However, <strong>more control is <em>not</em> always better</strong> — including <em>bad controls</em> can introduce additional bias.</p><p>This post revisits the bad control problem using a classical example and then explores it through the lens of <strong>Bayesian networks</strong>, including concepts like <strong>d-separation</strong> and the <strong>Markov blanket</strong>, to offer an intuitive understanding of how conditioning on the wrong variables can mislead your analysis.</p><h4>What are bad controls?</h4><p>Simple as it is, bad controls are variables that are themselves outcome variables in the notional experiment at hand.</p><h4>How bad controls bring additional biases?</h4><p>Short answer: because including bad controls will introduce selection bias.</p><h4>Intuitively, why will bad controls introduce selection bias?</h4><p>Consider this thought experiment: You randomly assign treatment and control groups, and at that point, they are well balanced. Now suppose that in your analysis, you condition on a variable Z — for example, by restricting attention to cases where Z=3. If Z is itself an outcome variable, this conditioning can disrupt the initial balance between the groups. In doing so, you introduce selection bias — this is the essence of a <em>bad control</em>.</p><h4>How to formally show the bad control problem in the equation?</h4><p>Example: Collider Bias</p><p>Suppose you are studying: <strong><em>Does smoking cause heart disease?</em></strong></p><p>The causal diagram (DAG Structure) is:</p><pre>Smoking (S) → # of Hospital Visit (H) ← Heart Disease (D)</pre><ul><li>Smoking increases the chance of going to the hospital (e.g., due to cough or lung issues).</li><li>Heart disease obviously also increases hospital visits.</li><li>But in the <strong>general population</strong>, smoking and heart disease <strong>may be independent</strong> (i.e., no causal relationship initially assumed).</li></ul><p>You are trying to estimate the effect of smoking on heart disease:</p><pre>E[D_i∣S_i=1]−E[D_i∣S_i=0]</pre><p>If you conditional on hospital visit =3, you are estimating:</p><pre>E[D_i∣S_i=1,H_i=3]−E[D_i∣S_i=0,H_i=3]</pre><p>If you further decompose, you get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFzoCYOJkYuWrSnSqE5XIA.png" /></figure><p>To see why the selection bias term is not zero</p><h3>E[D₀ᵢ | Sᵢ = 1, Hᵢ = 3]</h3><blockquote><em>This is the </em><strong><em>average disease level</em></strong><em> smokers </em><strong><em>would have had if they had not smoked</em></strong><em>, but </em><strong><em>only among those who actually visited the hospital 3 times</em></strong><em>.</em></blockquote><ul><li>We’re looking at people who <strong>did smoke</strong> and <strong>did go to the hospital</strong>.</li><li>But we’re asking a <strong>what-if</strong> question: <em>What if those same people had not smoked?</em></li><li>This term represents their <strong>baseline risk of disease</strong>, had they not received the treatment (i.e., smoking), <strong>within the hospital population</strong>.</li></ul><h3>E[D₀ᵢ | Sᵢ = 0, Hᵢ = 3]</h3><blockquote><em>This is the </em><strong><em>actual average disease level</em></strong><em> among </em><strong><em>non-smokers who visited the hospital 3 times</em></strong><em>.</em></blockquote><ul><li>These are people who <strong>did not smoke</strong>, and <strong>also visited the hospital</strong>.</li><li>Their disease outcome is observed directly since they never smoked.</li></ul><p>Clearly, these two groups of patients are very different and that’s why the selection bias term is not zero.</p><p>The sections above introduce the concept of bad controls and illustrate the issue with a classic example involving collider bias. For a more detailed discussion, see Chapter 3, Section 3.2.3 of <em>Mostly Harmless Econometrics</em> by Angrist and Pischke. Next, we will explore the bad control problem through the lens of Bayesian networks to develop a deeper, more intuitive understanding of why conditioning on the wrong variables can lead to biased estimates.</p><h3>Connection with Bayesian Network?</h3><h4>What is a Bayesian Network?</h4><p>A <strong>Bayesian network</strong> is a probabilistic graphical model that represents a set of variables and their conditional dependencies using a <strong>Directed Acyclic Graph (DAG)</strong>.</p><ul><li><strong>Nodes</strong> = variables (e.g., age, income, disease)</li><li><strong>Edges (arrows)</strong> = probabilistic dependencies (e.g., smoking → cancer)</li></ul><p>In short, it’s a <strong>map of probabilistic relationships</strong> that allows for <strong>efficient reasoning and inference</strong>. Causal inference uses similar DAGs but interprets the arrows <strong>causally</strong>, not just statistically.</p><h4>What is Markov Blanket?</h4><p>The <strong>Markov blanket</strong> of a node is the minimal set of variables you need to condition on to block all indirect paths of influence. For a node X, its Markov blanket includes:</p><ul><li>Its <strong>parents</strong> (direct causes)</li><li>Its <strong>children</strong> (direct effects)</li><li>Its children’s <strong>other parents</strong> (co-parents)</li></ul><p>In a Bayesian network, the Markov blanket of a node shields it from the rest of the network. If you condition on the blanket, nothing else can give you more information about that node. In other words, the Markov blanket tells you what you condition on.</p><p>In causal inference, this idea is central. When adjusting for confounding, <strong>we want to condition on variables that block backdoor (non-causal) paths</strong> without accidentally opening collider paths.</p><h4>What is d-Separation?</h4><p><strong>D-Separation</strong> is a graphical criterion to determine whether two variables are conditionally independent in a <strong>causal graph</strong> (a DAG). In simple terms, it tells us whether a path of potential information flow between two variables is blocked, given what we condition on.</p><p>There are three basic structures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ReIXWs-NCjX4cdrN1WV2Lw.png" /></figure><h4>How These Fit Together in Causal Inference</h4><p>Causal inference often rests on the assumption of <strong>conditional independence</strong>: that, once we control for the right variables, we can isolate the causal effect of a treatment or intervention. But if we control for the wrong variables (e.g., colliders), we may violate this assumption. Below table summarizes the connection and mapping between Causal Inference and Bayesian Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8LSADLAMKxyAyTspOU6-w.png" /></figure><h4>Takeaways for Practice and Interview</h4><ul><li>Don’t blindly include all variables in your regression or matching model. Be careful about what you condition on: sometimes <em>less is more</em>.</li><li>Use tools like d-separation and the Markov blanket to reason about dependency and independence.</li></ul><h3>Reference:</h3><p>Angrist, Joshua D., and Jörn-Steffen Pischke. <em>Mostly harmless econometrics: An empiricist’s companion</em>. Princeton university press, 2009.</p><p>Russell, Stuart J., and Peter Norvig. <em>Artificial intelligence: a modern approach</em>. pearson, 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9feb3af6ab54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Gibbs Sampling: Ergodicity, Detailed Balance, and Convergence]]></title>
            <link>https://medium.com/@xwang222/demystifying-gibbs-sampling-ergodicity-detailed-balance-and-convergence-ccd67b236cc0?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccd67b236cc0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[markov-chains]]></category>
            <category><![CDATA[gibbs-sampling]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 18:25:24 GMT</pubDate>
            <atom:updated>2025-02-28T18:25:24.967Z</atom:updated>
            <content:encoded><![CDATA[<p>Markov Chain Monte Carlo (MCMC) methods, such as Gibbs sampling, are widely used in Bayesian inference and probabilistic modeling. However, many learners — including myself — have experienced situations where we follow an online tutorial, run the code, and see that it <strong>converges to the true posterior</strong>. But then comes the realization: <em>Wait, what did I actually do?</em></p><p>It is easy to find explanations of <strong>how</strong> Gibbs sampling works, but understanding <strong>why</strong> it works is a different challenge. Learners often struggle with questions like:</p><ul><li><em>Why does Gibbs sampling actually work?</em></li><li><em>What guarantees that it converges to the correct posterior?</em></li></ul><p>This post breaks down these concepts step by step to help solidify these ideas.</p><h3>1. The High-Level Roadmap</h3><p>Gibbs sampling constructs a <strong>Markov Chain</strong> that updates one variable at a time. When you run Gibbs sampling, you are actively <strong>building this Markov Chain</strong>,</p><ul><li>where the sequence of sampled states forms a trajectory through the state space.</li><li>Each transition follows a specific rule: <strong>selecting one variable and sampling it from its conditional distribution</strong> while keeping the others fixed.</li></ul><p>For the Markov chain to <strong>converge to the true posterior</strong>, it must satisfy two key properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*uG-za5rPhteFONLKZLEijg.png" /></figure><p>That is, if a Markov chain is <strong>ergodic</strong> and satisfies <strong>detailed balance</strong>, it is <strong>guaranteed to converge</strong> to the correct stationary distribution.</p><h3>2. Ergodicity = Irreducibility + Aperiodicity</h3><p>For a Markov chain to be <strong>ergodic</strong>, it must be able to <strong>explore the full state space without getting trapped</strong> in certain regions or cycles. This requires two conditions:</p><h4>Irreducibility: Ensuring All States Are Reachable</h4><p>A Markov chain is <strong>irreducible</strong> if <strong>every state can be reached from any other state</strong> in a finite number of steps.</p><p>✅ <strong>How to ensure irreducibility?</strong></p><ul><li>All <strong>conditional probabilities must be strictly positive</strong> for valid transitions.</li><li>The state space must be <strong>fully connected</strong>, meaning there are no isolated clusters of states.</li></ul><h4>Aperiodicity: Avoiding Cyclic Behavior</h4><p>A Markov chain is <strong>aperiodic</strong> if it <strong>does not get stuck in cycles</strong> and has no fixed step size for returning to a state.</p><p>✅ <strong>How to ensure aperiodicity?</strong></p><ul><li>The chain should have <strong>some probability of staying in the same state</strong> at each step.</li><li>Avoid strict update cycles by <strong>randomizing the order of variable updates</strong>.</li></ul><p>For <strong>Gibbs sampling to work correctly</strong>, the <strong>transition kernel must satisfy both irreducibility and aperiodicity</strong> to ensure ergodicity.</p><h3>3. Detailed Balance</h3><p>Detailed balance ensures that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*MckHEIzEr92cNMqzsvhI7A.png" /><figcaption>Russell, S. J., &amp; Norvig, P. (2021). Artificial Intelligence: A Modern Approach, 4th, global edition Pearson.</figcaption></figure><p>This means that the <strong>flow of probability between any two states is symmetric</strong>, ensuring that the stationary distribution remains unchanged.</p><p>✅ <strong>Why does Gibbs sampling satisfy detailed balance?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*f83cWbyAEKDk756TXAaQ1w.png" /></figure><p>The above derivation serves to illustrate the core idea. For a more rigorous mathematical treatment, please refer to the materials listed in the reference section.</p><p>Since <strong>detailed balance</strong> holds by construction, Gibbs sampling is guaranteed to <strong>correctly sample from the true posterior</strong> — provided that the Markov chain is also <strong>ergodic</strong>.</p><h3>3. Python Demonstration</h3><pre>import numpy as np<br>import pandas as pd<br>from collections import Counter<br><br># Define the derived conditional probability functions from the joint distribution<br>def P_X1_given_X2_X3(x2, x3):<br>    if x2 == 0 and x3 == 0:<br>        return 0.5<br>    elif x2 == 0 and x3 == 1:<br>        return 0.5<br>    elif x2 == 1 and x3 == 0:<br>        return 0.6667<br>    elif x2 == 1 and x3 == 1:<br>        return 0.8182<br><br>def P_X2_given_X1_X3(x1, x3):<br>    if x1 == 0 and x3 == 0:<br>        return 0.5<br>    elif x1 == 0 and x3 == 1:<br>        return 0.5<br>    elif x1 == 1 and x3 == 0:<br>        return 0.6667<br>    elif x1 == 1 and x3 == 1:<br>        return 0.8182<br><br>def P_X3_given_X1_X2(x1, x2):<br>    if x1 == 0 and x2 == 0:<br>        return 0.6667<br>    elif x1 == 0 and x2 == 1:<br>        return 0.6667<br>    elif x1 == 1 and x2 == 0:<br>        return 0.6667<br>    elif x1 == 1 and x2 == 1:<br>        return 0.8182<br><br># Gibbs Sampling: update one variable at a time (random scan variant)<br>def gibbs_sampling(iterations=100000, burn_in=2000):<br>    # Initialize randomly<br>    x1, x2, x3 = np.random.choice([0, 1], size=3)<br>    samples = []<br>    variables = [&#39;x1&#39;, &#39;x2&#39;, &#39;x3&#39;]<br>    <br>    for i in range(iterations):<br>        var = np.random.choice(variables)<br>        if var == &#39;x1&#39;:<br>            p = P_X1_given_X2_X3(x2, x3)<br>            x1 = np.random.choice([0, 1], p=[1 - p, p])<br>        elif var == &#39;x2&#39;:<br>            p = P_X2_given_X1_X3(x1, x3)<br>            x2 = np.random.choice([0, 1], p=[1 - p, p])<br>        else:  # update x3<br>            p = P_X3_given_X1_X2(x1, x2)<br>            x3 = np.random.choice([0, 1], p=[1 - p, p])<br>        <br>        if i &gt;= burn_in:<br>            samples.append((x1, x2, x3))<br>    return samples<br><br># Run the Gibbs sampler<br>samples = gibbs_sampling()<br><br># Estimate the joint distribution from the samples<br>sample_counts = Counter(samples)<br>total = sum(sample_counts.values())<br>estimated_joint = {state: count/total for state, count in sample_counts.items()}<br><br># Convert to DataFrame for display<br>df_estimated = pd.DataFrame(<br>    [(s[0], s[1], s[2], prob) for s, prob in estimated_joint.items()],<br>    columns=[&quot;X1&quot;, &quot;X2&quot;, &quot;X3&quot;, &quot;Estimated P(X1, X2, X3)&quot;]<br>)<br><br>print(df_estimated.sort_values(by=[&quot;X1&quot;, &quot;X2&quot;, &quot;X3&quot;]).reset_index(drop=True))<br><br>&#39;&#39;&#39;<br>   X1  X2  X3  Estimated P(X1, X2, X3)<br>0   0   0   0                 0.051102<br>1   0   0   1                 0.100337<br>2   0   1   0                 0.050857<br>3   0   1   1                 0.100010<br>4   1   0   0                 0.050806<br>5   1   0   1                 0.098520<br>6   1   1   0                 0.100643<br>7   1   1   1                 0.447724<br>&#39;&#39;&#39;</pre><p>If you print out the samples, you will get the Markov chain you just built, where only one variable was updated from state to state, following the conditional distribution.</p><pre>print(samples)<br><br>&#39;&#39;&#39;<br>[(0, 1, 0), (1, 1, 0), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), <br>(1, 1, 1), (1, 1, 1), (1, 1, 0), (1, 1, 0), (1, 1, 1), (1, 1, 1), <br>(1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), <br>(1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), <br>(1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 1, 1), (1, 0, 1), (1, 0, 1), ...]<br>&#39;&#39;&#39;</pre><h3>4. Discussion and Thoughts</h3><p>Beyond our toy example, applying <strong>Gibbs sampling</strong> in real-world problems comes with significant challenges. In such cases, we often need to consider alternative <strong>MCMC methods</strong> that are better suited to the problem at hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*Ue0daCcomWx0xV9EggdzWg.png" /></figure><p>This post serves as an <strong>introduction or a conceptual foundation</strong> for understanding MCMC methods, particularly Gibbs sampling. My goal was to share the <strong>key concepts</strong> that helped me make sense of how and why Gibbs sampling works.</p><p>Bayesian statistics can be complex, with topics like <strong>Markov blankets</strong> and <strong>Bayesian networks</strong> requiring deeper study. If you’re interested in going further, I encourage you to <strong>explore textbooks and research papers</strong> to gain a more comprehensive understanding.</p><h3>Reference</h3><p>Russell, S. J., &amp; Norvig, P. (2021). Artificial Intelligence: A Modern Approach, 4th, global edition Pearson. <em>Harlow, UK</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccd67b236cc0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monotonic Constraints in ML: Missteps, Pitfalls, and Remedies]]></title>
            <link>https://medium.com/@xwang222/monotonic-constraints-in-ml-missteps-pitfalls-and-remedies-53182d87cea8?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/53182d87cea8</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[model-explainability]]></category>
            <category><![CDATA[xgboost]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Sun, 16 Feb 2025 22:10:43 GMT</pubDate>
            <atom:updated>2025-02-21T20:11:18.757Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In many machine learning applications — particularly in <strong>finance, economics, and risk modeling</strong> — we often have <strong>a priori knowledge</strong> that certain features should exhibit a monotonic relationship with the target variable.</p><p>For example, in a <strong>loan approval model</strong>, we intuitively expect that:<br>- <strong>Higher income</strong> should <strong>increase</strong> the likelihood of approval.<br>- <strong>Higher debt</strong> should <strong>decrease</strong> the likelihood of approval.</p><p>Enforcing these relationships through <strong>monotonic constraints</strong> in XGBoost can improve model <strong>interpretability</strong> and ensure <strong>regulatory compliance</strong> in high-stakes environments.</p><p>While implementing <strong>monotonic constraints</strong> in XGBoost is relatively straightforward (If you are not sure, please see my code link at the end of the post), <strong>determining the correct direction</strong> is where things get tricky. Many practitioners unknowingly fall into common pitfalls, relying on methods that don’t truly reflect <strong>intrinsic feature-target relationships</strong>.</p><p>This post will explore:<br>🔹 <strong>Common mistakes</strong> when determining monotonic constraints<br>🔹 <strong>Challenges in high-dimensional settings</strong><br>🔹 <strong>A new, systematic approach</strong> to automate the process</p><h3>Common Mistakes to Avoid</h3><h4><strong>Using PDPs or SHAP Dependence Plots for Monotonicity Direction</strong></h4><p>PDPs (<strong>Partial Dependence Plots</strong>) and SHAP Dependence Plots are <strong>powerful tools</strong> for understanding <strong>model behavior</strong>. So, it’s tempting to use them to determine the direction of monotonic constraints. However, <strong>this is a mistake</strong>.</p><p><strong>Why?</strong> These tools are <strong>model-dependent</strong> — they reflect relationships <strong>after the model is trained</strong>, rather than the <strong>true underlying trends in the raw data</strong>.</p><p><strong>Better alternative?</strong> Instead of relying on post-training artifacts, we should <strong>analyze raw feature-target relationships</strong> directly.</p><h3>How to Determine the Direction of Monotonicity</h3><ol><li><strong>Domain Knowledge</strong></li></ol><ul><li>Use business rules and logical relationships to decide constraints.</li><li>Example: Income <strong>(should increase)</strong> loan approval likelihood → <strong>+1</strong></li><li>⚠ Limitations: Some relationships are <strong>not obvious</strong>, making it difficult to determine the correct constraint <strong>purely from intuition</strong>.</li></ul><p><strong>2. Graph-Based Checks (Data-Driven Methods)</strong></p><ul><li>Use <strong>scatter plots, line charts, or binned averages</strong> to visualize the relationship between a feature and the target variable.</li><li>⚠ <strong>Limitation</strong>: Imagine manually inspecting <strong>200+ variables</strong> — your <strong>decisions will likely change</strong> throughout the process, introducing <strong>inconsistency and bias</strong>.</li></ul><h3>How to Address these Challenges?</h3><p>Given the limitations of manual inspection, a more robust approach is to <strong>systematically evaluate monotonicity</strong> using multiple statistical techniques in an <strong>automated</strong> manner.</p><p>Before diving deeper, it is important to acknowledge that determining the direction of monotonic constraints is ultimately an <strong>expert judgment</strong> — there is no universally correct answer in all cases. Automation is not intended to replace human intuition but rather to provide <strong>data-driven insights</strong> that help structure decision-making, while still requiring <strong>manual review</strong> for ambiguous cases.</p><p>The method I propose is <strong>exploratory, not flawless</strong>. It is designed to bring <strong>consistency</strong> to an inherently subjective process, serving as a <strong>starting point for refinement</strong> rather than a definitive rule.</p><h3>A More Scalable Approach</h3><p>Here’s how my approach works:</p><h4>Step 1: Assessing Overall Monotonicity</h4><ul><li>Compute <strong>Spearman’s correlation</strong> to get an initial estimate of monotonicity direction.</li><li>Perform <strong>quadratic regression</strong> to check for <strong>U-shaped</strong> or <strong>inverted U-shaped</strong> patterns.</li></ul><h4>Step 2: Weighting for Data-Dense Regions</h4><ul><li>Instead of treating all data points equally, weight them based on <strong>density</strong> using <strong>histogram binning</strong>. This ensures that <strong>regions with more observations</strong> contribute more to the monotonicity decision.</li></ul><h4>Step 3: Applying Multiple Methods to Check for Consistency</h4><ul><li><strong>Bidirectional Isotonic Regression</strong>: Allows for a <strong>change point</strong>, capturing scenarios where monotonicity <strong>reverses direction</strong>.</li><li><strong>OLS Regression</strong>: Checks for a <strong>significant slope</strong>.</li><li><strong>Piecewise Linear Regression</strong>: Detects <strong>slope changes</strong> to avoid misclassification.</li></ul><h4>Step 4: Robust Decision-Making</h4><p>Given the complexity of real-world data, relying on <strong>a single test or assumption</strong> is prone to error. Instead, my approach <strong>integrates</strong> the above methods to make a final decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*CmOQPgSx19rucBVtk4qd3Q.png" /></figure><h3>Experiments</h3><p>To verify the effectiveness of this approach, we tested it on <strong>synthetic datasets</strong> covering a wide range of real-world scenarios:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tv8hYQ1O4KXP80OkS-hXaA.png" /></figure><p>Below are the results. The texts in the parenthesis of the title are the final decision from my automated approach and it seems that my approach is quite robust:</p><ul><li>The approach correctly classified monotonicity in all structured cases.</li><li>It successfully avoided false U-shape detection in noisy data.</li><li>Uneven distributions did not mislead classification decisions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RpLLSzO9Zs7PpRGN0Y6-oQ.jpeg" /></figure><h3>Final Thoughts</h3><p>As seen in the experiments above, certain cases — such as the <strong>step function</strong> and the <strong>unevenly distributed data</strong> in the third row — highlight the nuances of monotonicity classification. One could argue that the step function exhibits a <strong>U-shape</strong>, while the second case might be considered <strong>monotonically increasing</strong>. Also, one could argue that giving more weights to the dense region might overlook the global pattern. Ultimately, the final decision hinges on <strong>domain knowledge, business context, and experience</strong>, reinforcing the importance of expert judgment in applying monotonic constraints.</p><p><strong>When to Use Monotonic Constraints:</strong></p><ul><li>When domain knowledge strongly supports a directional effect.</li><li>In regulated industries where explainability is critical.</li><li>When model interpretability is as important as accuracy.</li></ul><p><strong>Best Practices:</strong></p><ul><li>Use <strong>domain knowledge</strong> and <strong>graph-based checks</strong> to verify simple relationships.</li><li>Avoid <strong>model-dependent diagnostics</strong> (PDP, SHAP) for setting constraints.</li><li>Leverage <strong>automated trend scoring</strong> to scale monotonicity determination for large datasets.</li></ul><p>Code for hyperparameter tuning with monotonic constraints is <a href="https://github.com/xwang222/My-Projects/blob/main/Data%20Science/Hyerparameter%20Tuning%20with%20Monotonic%20Constraints.ipynb">here</a>.</p><p>If there’s strong interest or if I see significant engagement on this post, I’ll be happy to share the code for my exploratory approach.</p><p>Hope you enjoyed my post!</p><h3>Most Read Posts by My Audience</h3><ol><li><a href="https://medium.com/@xwang222/shap-value-dilution-how-xgboost-feature-sampling-misleads-0a20eed2e6c8">SHAP Value Dilution: How XGBoost Feature Sampling Misleads</a></li><li><a href="https://medium.com/p/446a67caf336">Causal Inference 102 EP04: Implementation of Inverse Probability Weighting</a></li><li><a href="https://medium.com/p/c0d02ce2f7bb">Causal Inference 101 EP02: Directed Acyclic Graph (DAG)</a></li></ol><h3>Reference</h3><ol><li><a href="https://xgboost.readthedocs.io/en/stable/tutorials/monotonic.html">https://xgboost.readthedocs.io/en/stable/tutorials/monotonic.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53182d87cea8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Importance Dilution in XGBoost: Just Like SHAP or Different?]]></title>
            <link>https://medium.com/@xwang222/feature-importance-dilution-in-xgboost-just-like-shap-or-different-274714987aea?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/274714987aea</guid>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[shap-values]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-importance]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Sat, 08 Feb 2025 14:40:48 GMT</pubDate>
            <atom:updated>2025-02-08T14:42:16.372Z</atom:updated>
            <content:encoded><![CDATA[<p>Gain-based feature importance is another popular metric to evaluate in machine learning models for both variable reduction and model explainability. In a previous <a href="https://medium.com/@xwang222/shap-value-dilution-how-xgboost-feature-sampling-misleads-0a20eed2e6c8">post</a>, we explored how <strong>SHAP values get diluted</strong> when highly correlated features exist and colsample_bytree &lt; 1.0. This raises an interesting question:</p><ul><li><strong>Does feature sampling (</strong><strong>colsample_bytree) also cause dilution in XGBoost’s built-in gain-based feature importance?</strong></li><li><strong>If so, does this dilution behave the same way as it does for SHAP values?</strong></li></ul><p>Let’s find out! <em>If you like my post, please follow me for more! Thanks!</em></p><h3>What We already Know</h3><p>(If these concepts are new to you, check out my previous <a href="https://medium.com/@xwang222/shap-value-dilution-how-xgboost-feature-sampling-misleads-0a20eed2e6c8">post</a>!)</p><ul><li><strong>Without feature sampling (</strong><strong>colsample_bytree = 1.0)</strong>, XGBoost consistently picks the <strong>first occurrence</strong> of a feature for splits, assigning all SHAP values to it.</li><li><strong>With feature sampling (</strong><strong>colsample_bytree &lt; 1.0)</strong>, some trees will <strong>exclude the first occurrence</strong>, forcing XGBoost to rely on its duplicate instead.</li></ul><h3>What We Expected to See</h3><ul><li><strong>Feature importance should also be diluted</strong> because information gain is affected by the same feature sampling mechanism.</li><li><strong>The dilution pattern in feature importance should behave similarly to SHAP values.</strong></li></ul><p>However, our experiments revealed an unexpected <strong>difference between SHAP dilution and feature importance dilution</strong>. Let’s dive into the results!</p><h3>Experimental Setup</h3><p>Consider a case where we train an XGBoost model using a dataset with 31 features, one of which is a <strong>duplicate of another (100% correlated)</strong>. We train the model multiple times, adjusting colsample_bytree and see the impact on the both SHAP values and feature importance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1qKJ185fntZsB1op7PPEw.png" /></figure><p>To summarize the findings, we see that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*be38a9RY0U9X4ArXaOxunQ.png" /><figcaption>Note: the above feature importance is based on the average gain. The total gain gives the same pattern.</figcaption></figure><p>But, why?</p><h3>1️⃣Feature Sampling might Inflate Information Gain</h3><p>XGBoost’s <strong>feature importance</strong> is derived from <strong>information gain</strong> at each split. When colsample_bytree is <strong>low</strong>, each tree sees <strong>fewer</strong> features, making it rely more heavily on the ones available. As a result, e<strong>ach split might look more valuable</strong> than it would in a full feature set, causing the <strong>gain per split to increase artificially.</strong></p><h3>2️⃣Why SHAP Values Show Smoother Dilution than Gain-Based Feature Importance?</h3><p>After reading through multiple references, this is what I figured out:</p><h4>🔹Information Gain is Tree-Specific</h4><ul><li>XGBoost <strong>builds each tree independently</strong>.</li><li>If a feature is selected for a split, all information gain is attributed to <strong>that feature</strong> in that tree.</li><li>If the duplicate feature (Feature_dup) is <strong>never selected</strong>, it gets <strong>zero</strong> importance.</li><li><strong>No dilution occurs because information gain is not shared across trees.</strong></li></ul><h4>🔹SHAP Redistributes Importance Across All Trees</h4><p>SHAP works differently because it c<strong>onsiders all trees in the ensemble</strong> and distributes feature importance across them. Consider the following example, where:</p><ul><li><strong>Tree 1</strong> uses Feature_A among other attributes</li><li><strong>Tree 2</strong> uses Feature_A_dup among other attributes</li></ul><p>This is what happens:</p><ol><li>When calculating the SHAP value, it compares the actual model score with a counterfactual model score where Feature_A is missing.</li><li><strong>What does it mean when a feature is “missing”?</strong></li></ol><ul><li>In <strong>TreeSHAP</strong>, if Feature_A is missing, it means we don’t know whether to take the left or right branch when we traverse the tree.</li><li>In that case, <strong>we simply take both paths and compute a weighted average</strong> of the leaf node outputs.</li><li>It follows that in our example, the counterfactual model score is a weighted average of some leaf nodes from <strong>both Tree 1 and Tree 2</strong>.</li><li>The SHAP value is then distributed <strong>between Tree 1 and Tree 2</strong> based on these weights.</li><li><strong>Even if Tree 2 never used </strong><strong>Feature_A, it still inherits some importance due to this redistribution.</strong></li><li>This means <strong>SHAP values “borrow” importance from other trees</strong>, leading to <strong>a smoother redistribution effect</strong>.</li></ul><h3>Thoughts and Discussions</h3><p>The above experiment (code is <a href="https://github.com/xwang222/My-Projects/blob/main/Data%20Science/Feature%20Importance%20Dilution.ipynb">here</a>) highlights several critical insights:</p><p>✅ <strong>Feature importance can be unstable</strong> when feature sampling is enabled.<br>✅ Using only XGBoost’s feature importance for interpretation may lead to misleading rankings.<br>✅ If you rely on feature importance for <strong>variable selection</strong>, consider <strong>disabling</strong> feature sampling (colsample_bytree=1.0).<br>✅ Consider both SHAP values and gain-based feature importance to get a more complete picture of feature relevance.</p><p>If you have different insights to share, please leave your comments below and we can discuss!</p><p>Hope you enjoyed my post!</p><h3>Reference:</h3><ol><li><a href="https://shap.readthedocs.io/en/latest/example_notebooks/tabular_examples/tree_based_models/Python%20Version%20of%20Tree%20SHAP.html">https://shap.readthedocs.io/en/latest/example_notebooks/tabular_examples/tree_based_models/Python%20Version%20of%20Tree%20SHAP.html</a></li><li><a href="https://afiodorov.github.io/2019/05/20/shap-values-explained/">https://afiodorov.github.io/2019/05/20/shap-values-explained/</a></li><li><a href="https://arxiv.org/pdf/1802.03888">https://arxiv.org/pdf/1802.03888</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=274714987aea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SHAP Value Dilution: How XGBoost Feature Sampling Misleads]]></title>
            <link>https://medium.com/@xwang222/shap-value-dilution-how-xgboost-feature-sampling-misleads-0a20eed2e6c8?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a20eed2e6c8</guid>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[shap-values]]></category>
            <category><![CDATA[model-explainability]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Sun, 02 Feb 2025 04:24:36 GMT</pubDate>
            <atom:updated>2025-02-08T18:06:43.872Z</atom:updated>
            <content:encoded><![CDATA[<p>Interpreting machine learning models is critical in high-stakes applications like finance, healthcare, and fraud detection. <strong>SHAP (SHapley Additive Explanations)</strong> is one of the most reliable tools for explaining model behavior, helping us understand how features contribute to predictions.</p><p>However, if your XGBoost model <strong>uses feature sampling (</strong><strong>colsample_bytree) and has highly correlated features</strong>, SHAP values can be <strong>misleadingly diluted</strong>—potentially leading you to undervalue key features.</p><p>Let’s dive into why this happens, when you should be concerned, and how to interpret SHAP correctly in the presence of feature sampling.</p><p><em>If you find my post helpful and like it, please follow me! Thanks!</em></p><h3>Why Use Feature Sampling in XGBoost?</h3><p>Feature sampling (controlled by thecolsample_bytree) is an important technique in XGBoost that reduces overfitting, speeds up training, and improves diversity among trees in the ensemble. In fact, it is one of the popular hyperparameters tuned during the model development. However, <strong>it can distort SHAP values when highly correlated features exist</strong> — which is a problem if model interpretability is a priority.</p><h3>How Feature Sampling Affects SHAP Values</h3><p>When colsample_bytree = 1.0, each tree sees the full set of features, so the SHAP values will all be assign to the <strong>first occurrence</strong> of the correlated features.</p><p>However, when colsample_bytree &lt; 1.0, each tree sees only a subset of features:</p><ul><li>Sometimes, <strong>Feature A</strong> is included, and Feature B is missing.</li><li>Other times, <strong>Feature B</strong> is included, and Feature A is missing.</li><li>Their SHAP values get <strong>spread across multiple features</strong>, making both seem less important than they truly are.</li></ul><p>If interpretability is your goal, this dilution can lead to incorrect conclusions about <strong>which features matter most</strong>.</p><h3>Real-World Example: The Impact of colsample_bytree on SHAP</h3><p>Consider a case where we train an XGBoost model using a dataset with five features, one of which is a <strong>duplicate of another (100% correlated)</strong>.</p><p>We train the model three times, adjusting colsample_bytree and see the impact on SHAP values:</p><ul><li><strong>colsample_bytree = 1.0</strong> → All trees see all features, and SHAP values correctly reflect importance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*rnzlJU-v-HOxVx4O8xa0Lw.png" /></figure><ul><li><strong>colsample_bytree = 0.9</strong> → Some trees exclude a feature, slightly spreading SHAP values across correlated features.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*o9RrmIrGVCRshaioV7QlCQ.png" /></figure><ul><li><strong>colsample_bytree = 0.3</strong> → Only 30% of features are seen per tree, causing significant dilution in SHAP values.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*1HHgqdcu6MMCTnhxZlV1RA.png" /></figure><p><strong>Key Finding:</strong> When colsample_bytree is small, both correlated features have <strong>low SHAP values</strong>, even though they contain valuable predictive information.</p><h3>Root Cause of the Behavior of SHAP Values</h3><p>XGBoost greedily selects the best feature splits based on information gain. When two features are <strong>highly correlated</strong>, XGBoost <strong>prioritizes the first occurrence in the dataset</strong>, ignoring the duplicate. Assuming that we have two duplicated columns in our data set (thus, 100% correlation), this is what happens in the backend:</p><ul><li><strong>XGBoost scans features in order</strong> and evaluates potential splits based on the <strong>gain in information</strong> (such as reduction in impurity).</li><li>When two features (A and B) are <strong>perfectly correlated</strong>, they have <strong>identical split scores</strong>.</li><li>Since XGBoost <strong>traverses features in order</strong>, the first occurrence of the correlated features will always be chosen. (You can see my proof of concept in my code shared below.)</li><li>The second occurrence is <strong>never used</strong> because it provides no additional information after the first occurrence is split on.</li><li>When colsample_bytree &lt; 1.0, each tree sees only a subset of features. In cases where only feature B is selected, SHAP values will be assigned to it in those trees.</li><li>When colsample_bytree becomes smaller, there will be more trees with only B included, thus getting more SHAP values assigned, leading to significant dilution.</li></ul><p>Here is the <a href="https://github.com/xwang222/My-Projects/blob/main/Data%20Science/SHAP_Value_Dilution.ipynb">code</a> I used. If you are interested in, please check it out!</p><h3>Final Thoughts</h3><p>Feature sampling is a powerful technique for improving model generalization, but when <strong>highly correlated features exist</strong>, SHAP values can become <strong>diluted and misleading</strong>. If you want to see what happens to the gain-based feature importance, check out this <a href="https://medium.com/@xwang222/feature-importance-dilution-in-xgboost-just-like-shap-or-different-274714987aea">post</a>.</p><p>If you’re using SHAP to explain your model, especially in regulated or high-risk applications, make sure you <strong>understand how feature sampling influences feature importance.</strong> Otherwise, you might <strong>underestimate key drivers of your model’s decisions</strong> — leading to poor insights and suboptimal decision-making.</p><p>Hope you enjoyed my post!</p><h3>Most Read Posts by My Audience</h3><p>If you found this article insightful, you might also enjoy these popular reads:</p><p><a href="https://medium.com/p/446a67caf336">Causal Inference 102 EP04: Implementation of Inverse Probability Weighting</a></p><p><a href="https://medium.com/p/c0d02ce2f7bb">Causal Inference 101 EP02: Directed Acyclic Graph (DAG)</a></p><p><a href="https://medium.com/p/9f3a11fbb374">Forecasting 101 EP07: Multivariate Models</a></p><h3><strong>Reference:</strong></h3><ul><li><a href="https://shap.readthedocs.io/en/latest/example_notebooks/overviews/An%20introduction%20to%20explainable%20AI%20with%20Shapley%20values.html">An introduction to explainable AI with Shapley values - SHAP latest documentation</a></li><li><a href="https://xgboost.readthedocs.io/en/stable/">XGBoost Documentation - xgboost 3.3.0 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a20eed2e6c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No Hype, Just Results: DeepSeek Educated Me Better on Correlation and VIF]]></title>
            <link>https://medium.com/@xwang222/no-hype-just-results-deepseek-educated-me-better-on-this-f649ca2a288d?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f649ca2a288d</guid>
            <category><![CDATA[variable-reduction]]></category>
            <category><![CDATA[deepseek-r1]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[correlation]]></category>
            <category><![CDATA[variance-inflation-factor]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 05:13:24 GMT</pubDate>
            <atom:updated>2025-01-28T18:05:37.858Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of data science, having too many variables can be both a blessing and a curse. Doing variable reduction helps to improve model performance, simplify interpretability, and enhances efficiency.</p><p>Among the many techniques available, <strong>pairwise correlation</strong> and <strong>variance inflation factor (VIF)</strong> are two commonly used methods to filter out highly correlated variables. In practice, the rules of thumb often applied are <strong>pairwise correlation &lt; 90%</strong> and <strong>VIF &lt; 10</strong> as thresholds for acceptable multicollinearity. However, two key questions have always intrigued me:</p><ol><li><strong>Does having all VIF &lt; 10 imply all pairwise correlations are &lt; 90%?</strong></li><li><strong>Does having all pairwise correlations &lt; 90% imply all VIF &lt; 10?</strong></li></ol><p>Until …. I decided to explore these questions with the help of <strong>DeepSeek</strong> and <strong>ChatGPT-4o</strong>.</p><h3>Key Takeaways:</h3><ol><li><strong>Neither </strong>of the statements<strong> is true.</strong></li><li><strong>DeepSeek </strong>seems to <strong>educate me better </strong>on this question<strong>.</strong></li></ol><p>If you are interested in how DeepSeek and ChatGPT responded to these two questions, please read the rest of this post!</p><h3>Concepts:</h3><p><strong>Pairwise correlation</strong> measures the linear relationship between two variables, with values ranging from −1 (perfect negative correlation) to +1 (perfect positive correlation). When two variables are highly correlated (e.g., correlation &gt; 0.9), they convey similar information, making one of them redundant.</p><p><strong>Variance Inflation Factor (VIF)</strong> quantifies the degree of multicollinearity among variables by measuring how much the variance of a regression coefficient is inflated due to correlation with other variables. A VIF value<strong>&gt; </strong>10 usually indicates high multicollinearity (consider dropping the variable).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/0*iXAcuE8HlWZPQFaB" /></figure><p>where <em>R</em>2<em>i</em> is the coefficient of determination of the regression equation in step one, with Xi on the left hand side, and all other predictor variables (all the other X variables) on the right hand side.</p><h3>Asking ChatGPT-4o</h3><p>When I asked ChatGPT-4o these two questions, its responses fell short. As shown in the screenshot below, the model struggled to provide a concrete solution. While there might be some randomness to its performance, I tested it three times and received similar feedback each time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*7Xo203XB7r6QJoYV7cBifg.png" /></figure><h3>Asking DeepSeek R1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*zR8qEhv5C5GV14pXylnPbw.png" /></figure><p>I asked DeepSeek R1 and it came up with the example in the above screen copy. Sounds legitimate, but I don’t like edge cases like 0 correlation, so I asked DeepSeek to refine the answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*APGpIsMd4iXyHz55-fYtZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*-N6rww7AIL4215o48vPoDQ.png" /></figure><p>It didn’t come up with the right example at the first attempt, but it made a self correction and figured out a correct example (see screen copy above). Then, I continued with the second question:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*uFK1zQ3nsolUAjxp1zy-5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*lMU3Qgxq8tuIsN7WwtcilQ.png" /></figure><p>For the second question, DeepSeek’s approach was similarly robust. I verified the code it provided, and the results were accurate. You can find my notebook <a href="https://github.com/xwang222/My-Projects/blob/main/Data%20Science/Pairwise_Correlation_and_VIF.ipynb">here</a> if you want play with it!</p><p><strong>Good job, DeepSeek!</strong></p><h3>Thoughts and Discussion:</h3><p>With the help of DeepSeek, I learnt that:</p><ol><li>All VIF&lt;10<strong> DOES NOT </strong>imply all pairwise correlation&lt;90%.</li><li>All pairwise correlation&lt;90% <strong>DOES NOT </strong>imply all VIF&lt;10.</li><li>It seems that <strong>DeepSeek</strong> did a better job on this question.</li></ol><p>With tools like DeepSeek and ChatGPT-4o, learning has become more accessible, cost-efficient, and effective. While we may not all become the developers behind these advanced AI models, we can leverage them to learn efficiently and push the boundaries of our own understanding.</p><p>By learning from AI, we are closing the gap between where we are and where these technologies can take us — faster than ever before.</p><h3>Reference:</h3><ol><li><a href="https://en.wikipedia.org/wiki/Variance_inflation_factor">https://en.wikipedia.org/wiki/Variance_inflation_factor</a></li><li><a href="https://github.com/deepseek-ai/DeepSeek-R1">https://github.com/deepseek-ai/DeepSeek-R1</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f649ca2a288d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How XGBoost Handles Missing Values? A Proof of Concept]]></title>
            <link>https://medium.com/@xwang222/how-xgboost-handles-missing-values-a-proof-of-concept-6aa7afcc8eb9?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa7afcc8eb9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[missing-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[xgboost]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 22:29:29 GMT</pubDate>
            <atom:updated>2025-01-26T22:29:29.704Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>When working with machine learning, handling missing values in data is a common challenge. XGBoost, a widely used gradient boosting algorithm, has a unique way of dealing with missing values during training. This article will explain how XGBoost treats missing values and demonstrate the behavior under different scenarios.</p><p>I know you guys are all busy and want to see the conclusion right away, so I list the key takeaways right here:</p><h3>Key Takeaways:</h3><ol><li><strong>Adaptive Handling:</strong> When missing values exist in the training data, XGBoost dynamically learns the optimal branch (left or right) for these instances.</li><li><strong>Default Handling:</strong> If no missing values are in the training data, missing values during inference are sent to the right branch by default.</li></ol><p>If you are interested in the proof of concept, please jump to the <strong>proof of concept</strong> section directly. If you are interested in the details, please read the rest of this post!</p><h3>Are Null Values always a Bad thing?</h3><p>Handling missing data properly is essential in data science because missing values can directly affect the integrity and performance of machine learning models. Improper handling can lead to issues like model bias, information loss, overfitting, or underfitting.</p><p>However, there are situations where imputing special or default values (e.g., 99999) as Null might be beneficial. For example, when enhancing model interpretability by enforcing monotonicity constraints on predictors, imputing default values as Null can simplify the process.</p><p>Consider a variable like “years of education,” where valid values range from 0 to 25, and special flag values such as 999 or 998 indicate different types of missing data. If a positive monotonicity constraint is applied, leaving 999 as a numeric value would make the model incorrectly predict a higher value for 999 than for 25. This would misrepresent the underlying relationship and reduce the model&#39;s reliability.</p><h3>Potential Solutions:</h3><ol><li><strong>Group Flag Values</strong>: Assign flag values like 999 or 998 to either the minimum (0) or maximum (25) value. While effective, this approach can become time-consuming when dealing with numerous predictors.</li><li><strong>Impute as Null</strong>: A simpler alternative is to treat these default values as Null and let the machine learning algorithm determine the appropriate group for them. This approach leverages the algorithm’s built-in handling of missing data, saving time and effort.</li></ol><h3>Wait a Moment! Something is missing…</h3><p>A valid concern arises when a model trained without Null values encounters Nulls in production data. How does XGBoost handle this scenario? Well, in such cases, forget about XGBoost, your first steps should include:</p><ol><li><strong>Verify the Data Pipeline</strong>: Double-check your code to ensure the data ingestion process is functioning correctly.</li><li><strong>Consult Upstream Sources</strong>: Check with the data vendor or upstream systems to determine if variable construction methods have changed.</li><li><strong>Assess Model Redevelopment</strong>: Consider whether the model needs to be retrained with updated data to accommodate this new behavior.</li></ol><p>Going back to our previous discussion regarding XGBoost, up to this point, two questions remain unclear:</p><ol><li>How XGBoost handles Null values at the training stage?</li><li>How XGBoost models handle Null values if Null values didn’t appear in the training data?</li></ol><h3>Proof of Concept</h3><h3>Case 1: Training Data Contains Missing Values</h3><p>XGBoost is designed to handle missing values gracefully. <strong>During training, it learns which branch (left or right) to assign to instances with missing values at each split. </strong>This behavior allows the algorithm to make the best use of incomplete data.</p><p>To illustrate, we used the Breast Cancer dataset and intentionally introduced missing values into one of the features. After training an XGBoost model, we visualized one of the decision trees. In the tree plot, you will notice how XGBoost explicitly handles missing data, routing it to either the left or right branch based on what optimizes the model’s performance (see the purple box in the figure below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlhNHYk2t5ZziUdWNo0iSA.png" /></figure><h3>Case 2: Training Data Without Missing Values</h3><p>In this scenario, the dataset does not contain any missing values. XGBoost still reserves a default behavior for handling missing data. By design, if a missing value appears during inference or testing, it <strong>always goes to the right branch of a split</strong>. This ensures consistency and robustness in handling unexpected missing data during model deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMvI9bGsY1vImUR9DIYOrQ.png" /></figure><p>Here is the code I used to create these examples:</p><pre>import numpy as np<br>import pandas as pd<br>from xgboost import XGBClassifier, plot_tree<br>import matplotlib.pyplot as plt<br>from sklearn.model_selection import train_test_split<br>from sklearn.datasets import load_breast_cancer<br><br># Load the breast cancer dataset<br>def load_data(case):<br>    data = load_breast_cancer()<br>    X = pd.DataFrame(data.data, columns=data.feature_names)<br>    y = data.target<br><br>    if case == 1:<br>        # Introduce missing values in the training data<br>        X.iloc[100:1500, 0] = np.nan<br><br>    return X, y<br><br># Function to train and plot the tree<br>def train_and_plot(case):<br>    # Load data for the specific case<br>    X, y = load_data(case)<br><br>    # Split data into train and test sets<br>    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br>    # Train an XGBoost model<br>    model = XGBClassifier(use_label_encoder=False, eval_metric=&#39;logloss&#39;)<br>    model.fit(X_train, y_train)<br><br>    # Plot one of the trees<br>    plt.figure(figsize=(16, 12), dpi=600)<br>    plot_tree(model, num_trees=0)<br>    plt.title(f&quot;Case {case}: {&#39;With Missing Values&#39; if case == 1 else &#39;No Missing Values&#39;}&quot;)<br>    plt.savefig(f&quot;xgboost_tree_case_{case}.jpg&quot;, format=&quot;jpg&quot;, dpi=600)<br>    plt.show()<br><br># Case 1: Training data contains missing values<br>train_and_plot(case=1)<br><br># Case 2: Training data does not contain missing values<br>train_and_plot(case=2)</pre><p>Hope you guys enjoyed this post!</p><p><strong>References:</strong></p><ol><li><a href="https://xgboost.readthedocs.io/en/stable/faq.html">https://xgboost.readthedocs.io/en/stable/faq.html</a></li><li><a href="https://towardsdatascience.com/xgboost-is-not-black-magic-56ca013144b4">https://towardsdatascience.com/xgboost-is-not-black-magic-56ca013144b4</a></li><li><a href="https://stackoverflow.com/questions/77000845/xgboost-model-predicting-with-nan-input-values">https://stackoverflow.com/questions/77000845/xgboost-model-predicting-with-nan-input-values</a></li><li><a href="https://datascience.stackexchange.com/questions/15305/how-does-xgboost-learn-what-are-the-inputs-for-missing-values">https://datascience.stackexchange.com/questions/15305/how-does-xgboost-learn-what-are-the-inputs-for-missing-values</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa7afcc8eb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Causal Inference 102 EP04: Implementation of Inverse Probability Weighting]]></title>
            <link>https://medium.com/@xwang222/causal-inference-102-ep04-implementation-of-inverse-probability-weighting-446a67caf336?source=rss-efca2a3557f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/446a67caf336</guid>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Xwang]]></dc:creator>
            <pubDate>Fri, 06 Jan 2023 03:24:39 GMT</pubDate>
            <atom:updated>2023-01-06T03:24:39.325Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, I will demonstrate how to implement Inverse Probability Weighting (IPW) from scratch. For simplicity, I use the well-known Lalonde dataset for illustration purposes. The full notebook is <a href="https://github.com/xwang222/My-Projects/blob/main/Causal%20Inference/IPW/IPW_from_Scratch.ipynb">here</a>, which is modified based on <a href="https://gist.github.com/BioSciEconomist/826556977841cbf966f03542b2be0c55">this</a>.</p><ol><li><strong>Motivating IPW</strong></li></ol><p>If you are reading this post, you probably understand why a simple OLS seldom produces causal estimates. If you don’t know that, please read my post <a href="https://medium.com/@xwang222/causal-inference-101-ep01-why-do-we-care-7ad638ad3e4d">here</a>.</p><p>One of the popular criticisms of using simple OLS to find the causal effect is that the samples in the treatment group may not be comparable to those in the control group. However, this also implies that if we can show that the treatment group and control group are comparable, we can just use simple OLS to estimate the causal effect. IPW is one of those methods that try to establish the comparability between the treatment and control groups.</p><p><strong>2. What does IPW do?</strong></p><p>What IPW does is to construct a pseudo-population where the treatment group is or seems to be comparable to the control group. Here, comparable means the characteristics of samples in both groups are similar or <strong>balanced</strong>.</p><p>To do so, IPW first calculates the probability of being treated for each observation, which is known as the <strong>propensity score</strong>. To see why using the inverse of that probability as the weight can improve the balance between the treatment and control groups, let’s look at the below example.</p><p>Say, John and Bob were both in the treatment group with propensity scores equal to 0.9 and 0.4 respectively. That means, for some reason, John (in the <strong>treatment</strong> group) is very different from all samples in the <strong>control</strong> group because his characteristics grant him a very high probability of being treated (0.9 in this case). It follows that if we have more Johns in the treatment group, the characteristics of the treatment group will become more different from the control group, which goes to the opposite of our goal — We want to establish/argue that the treatment group is comparable to the control group.</p><p>On the other hand, Bob has a propensity score of 0.4, which means his characteristics didn’t really determine his treatment status, which means Bob is to some extent similar to samples in the control group. It also follows that if we have more Bobs in our treatment group, it is possible to make the treatment group comparable to those of the control group in terms of characteristics.</p><p>Combining the above two pieces of information, it is clear that we want fewer Johns and more Bobs in our sample (to improve the balance), which means we want a lower weight on John and a higher weight on Bob. Since we already have their propensity score, the inverse of the propensity score just meets the need: 1/0.9&lt;1/0.4, which means we can just use the inverse of the propensity score as the weight to improve the balance between the treatment and control groups. This is what IPW does exactly.</p><p><strong>3. Python Implementation</strong></p><p>There are existing packages in Python, R, or Stata that one can use to do the IPW in one shot, but I believe implementation from scratch will definitely enhance our understanding.</p><p>We first load our data and some necessary packages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*z9OBNf123W6NAgfYUs4Xaw.png" /></figure><p>Then, we first produce the propensity score. In this example, we use logit as our link function to obtain the propensity score:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*2Yqoh0bjRGHXxy3B6cMTNQ.png" /></figure><p>Now, let’s take a look at the balance between the two groups for the unweighted and weighted versions. We can see that after applying the weights, the means of the treatment group are closer to those of the control group, though not perfect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*N1wbOpyFKt5QJS40L7J-RA.png" /></figure><p>Now, let’s estimate the treatment effect. We start with the unweighted version. We see that OLS tells us that the treatment effect is -635.02 and statistically insignificant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*hkH7lPbvFR9RYeJ8Rn3E2A.png" /></figure><p>Now, let’s try the weighted version. To help you understand the math behind it, I implemented three approaches.</p><p><strong>Approach 1</strong>: Compared weighted means of each group directly</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*5HUn0GD7BcwXWYYniHsQTQ.png" /></figure><p><strong>Approach 2</strong>: Using Weighted Least Square Package</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*urprPRDjPUeEHon-N7Ihmg.png" /></figure><p><strong>Approach 3</strong>: Weighted Least Square from Scratch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*GnJM5LdTSE06sqeTSuA8kw.png" /></figure><p>I also showed how to obtain the t-statistics and the P-value from scratch in the above code. In summary, with IPW, the treatment effect on the treated is 1214 and statistically significant at the 5% level. One thing to keep in mind here is having a different coefficient from IPW doesn’t necessarily imply the OLS result is biased. <strong>We can only tell those two estimates are different</strong>, but if the weighting was inappropriate, the estimates from IPW can even be more biased.</p><p><strong>4. Some Additional Considerations on IPW</strong></p><p>In the above steps, I explained the idea of IPW and implemented it from scratch in Python. However, these are just for illustration purposes. More considerations are necessary before we apply this method to a real research project.</p><p><strong>4.1 Balance Check:</strong><br>In my demonstration above, I only looked at the mean of each group. However, having a similar mean doesn’t necessarily imply that the two groups are balanced. There are several additional tools/tests that we can use to possibly strengthen our argument about the balance, including the SMD plot, Two-sample T-test, and F-test. Though there are criticisms of using these methods to argue for balance, my understanding here is these checks are necessary conditions for having balanced characteristics between the treatment and control groups. In order not to make this post too long, I will save the detailed implementation for a later post.</p><p><strong>4.2 Outliers — Stabilized IPW+Propensity Score Trimming</strong></p><p>Remember for IPW, the weights come from the inverse of the propensity score. Since it is totally reasonable to get propensity scores that are very close to 1 or 0, it follows that the corresponding weight can be huge. To address this concern, we can either use the normalized weight (called stabilized IPW) or trim the propensity score before taking the inverse (which just drops those outliers). For a detailed discussion, please see this material <a href="http://www2.stat.duke.edu/~fl35/teaching/640/Chap3.4_observational_weighting.pdf">here</a>.</p><p><strong>4.3 Compared with PSM (Propensity Score Matching)</strong></p><p>To see the difference in the most straightforward way, I would rename IPW as Inverse Proenpsity Score Weighting. Clearly, PSM is to find the matching samples in the other group while IPW is to achieve the balance by using different weights on the samples. For a detailed discussion, please see this material <a href="http://www2.stat.duke.edu/~fl35/teaching/640/Chap3.4_observational_weighting.pdf">here</a>.</p><p><strong>5. Ending Words</strong></p><p>In this post, I have introduced and explained the idea of IPW and hopefully, you get the intuition. During my search and study on google, I found the below two reference materials super helpful and I suggest you read them as well. The <a href="https://ww2.amstat.org/meetings/biop/2020/onlineprogram/handouts/SC4-Handouts.pdf">first one</a> is an introduction to Causal Inference. This reference explains some basic concepts really well with graphs. The <a href="https://ms.mcmaster.ca/canty/teaching/stat3a03/Lectures7.pdf">second one</a> is more like technical documentation for Weighted Least Square.</p><p>I have developed a series of posts about causal inference, please check them here:</p><p><strong><em>Causal Inference 101</em></strong></p><p>-EP01: <a href="https://medium.com/@xwang222/causal-inference-101-ep01-why-do-we-care-7ad638ad3e4d">Why do We Care?</a><br>-EP02: <a href="https://medium.com/@xwang222/causal-inference-101-ep02-directed-acyclic-graph-dag-c0d02ce2f7bb">Directed Acyclic Graph (DAG)</a><br>-EP03: <a href="https://medium.com/@xwang222/causal-inference-101-ep03-conditioning-strategy-part-i-45320a4f2a55">Conditioning Strategy-Part I-Subclassification</a><br>-EP04: <a href="https://medium.com/@xwang222/causal-inference-101-ep04-conditioning-strategy-part-ii-cf174b18b46b">Conditioning Strategy-Part II-Matching</a><br>-EP05: <a href="https://medium.com/@xwang222/causal-inference-102-ep02-implementation-of-difference-in-difference-did-3276caca4989">Difference-in-Difference</a><br>-EP06: <a href="https://medium.com/@xwang222/causal-inference-ep06-panel-data-methods-857b24781805">Panel Data Methods</a><br>-EP07: <a href="https://medium.com/@xwang222/causal-inference-ep07-instrumental-variable-approach-d95c66f6a8a1">Instrumental Variable Approach</a><br>-EP08: <a href="https://medium.com/@xwang222/causal-inference-ep08-regression-discontinuity-design-rdd-9b9d7a5cd827">Regression Discontinuity Design</a><br>-EP09: <a href="https://medium.com/@xwang222/causal-inference-101-ep09-experiment-design-part-i-dd395b295768">Experiment Design — Part I</a><br>-EP10: <a href="https://medium.com/@xwang222/causal-inference-101-ep10-experiment-design-part-ii-bb12dc996352">Experiment Design — Part II</a></p><p><strong><em>Causal Inference 102</em></strong></p><p>-EP01: <a href="https://medium.com/@xwang222/causal-inference-102-ep01-implementation-of-propensity-score-matching-12df26f312d7">Implementation of Propensity Score Matching from Scratch</a><br>-EP02: <a href="https://medium.com/@xwang222/causal-inference-102-ep02-implementation-of-difference-in-difference-did-3276caca4989">Implementation of Difference in Difference from Scratch</a><br>-EP03: <a href="https://medium.com/@xwang222/causal-inference-102-ep03-power-analysis-from-scratch-dd7dcaac14b5">Power Analysis From Scratch</a><br>-EP04: <a href="https://medium.com/@xwang222/causal-inference-102-ep04-implementation-of-inverse-probability-weighting-446a67caf336">IPW From Scratch</a></p><p>If you find a mistake in this series or if you want me to explain a topic that is not covered, please leave a message and I get back to you as soon as possible.</p><p><strong>Reference:<br></strong><a href="https://ww2.amstat.org/meetings/biop/2020/onlineprogram/handouts/SC4-Handouts.pdf">https://ww2.amstat.org/meetings/biop/2020/onlineprogram/handouts/SC4-Handouts.pdf</a><br><a href="https://ms.mcmaster.ca/canty/teaching/stat3a03/Lectures7.pdf">https://ms.mcmaster.ca/canty/teaching/stat3a03/Lectures7.pdf</a><br><a href="http://www2.stat.duke.edu/~fl35/teaching/640/Chap3.4_observational_weighting.pdf">http://www2.stat.duke.edu/~fl35/teaching/640/Chap3.4_observational_weighting.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=446a67caf336" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>