<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jakob Runge on Medium]]></title>
        <description><![CDATA[Stories by Jakob Runge on Medium]]></description>
        <link>https://medium.com/@jakob_6124?source=rss-42868c07c23b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BQdveg8dLni4_6zK-gAuuw.jpeg</url>
            <title>Stories by Jakob Runge on Medium</title>
            <link>https://medium.com/@jakob_6124?source=rss-42868c07c23b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 20:05:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jakob_6124/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nonlinear Causal Effect Estimation with Python]]></title>
            <link>https://medium.com/causality-in-data-science/nonlinear-causal-effect-estimation-with-python-b4edfd8251a9?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4edfd8251a9</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Wed, 12 Mar 2025 13:31:54 GMT</pubDate>
            <atom:updated>2025-03-13T20:07:09.074Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vN7RqlOoCqdu2CDspiyEdg.png" /></figure><p>A Gentle Guide to Causal Inference with Machine Learning Pt. 11</p><p><em>Relationships aren’t always linear. Things can get complicated.</em></p><p><em>Huh? Are we talking about romantic relationships? No, not really, but funnily it appears this sentence has universal meaning. When talking about causal dependencies, we can think of plenty of cases in which the effect is not linearly dependent on its cause. This predominance of nonlinearity in the world of patterns is the primary reason for complex machine learning models to be so successful. If everything were linear, we wouldn’t need anything but linear classifiers and regressors, right?</em></p><p><em>So okay, things are often non-linear. But at the same time, simply learning arbitrary patterns, meaning spurious correlations, makes our model hardly trustworthy. Causal Machine Learning comes with the idea of solving this problem. But how? You will find an answer to this question in the following blog post.</em></p><p><em>Before you keep on reading, please make sure you are familiar with the following concepts:</em></p><ol><li>C<a href="https://medium.com/causality-in-data-science/assumptions-for-causal-discovery-cc194d607a14"><em>ausal Markov, faithfulness and sufficiency assumption</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-is-causal-machine-learning-ceb480fd2902"><em>The fundamental idea behind causal inference</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a"><em>Causal graphs</em></a><em> &amp; d</em><a href="https://medium.com/causality-in-data-science/confounding-colliding-d-separation-and-sleeping-with-shoes-on-8ba43c976354"><em>-separation</em></a></li><li><a href="https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c"><em>The PC algorithm</em></a></li><li><a href="https://medium.com/causality-in-data-science/why-machine-learning-needs-causality-3d33e512cd37"><em>The reason why machine learning can profit from causality</em></a></li><li><a href="https://medium.com/causality-in-data-science/hands-on-causal-effect-estimation-with-python-aac40ca2cae0"><em>Causal Effect Estimation</em></a></li></ol><h4>A Quick Recap on Causal Inference and Machine Learning:</h4><p>As with all machine learning applications, causal machine learning in the end boils down to a model that should live up to certain desirable standards.</p><p>While in usual machine learning this desirability is defined by some metrics like accuracy, causal machine learning desirability goes beyond “only” delivering good predictions. It should also deliver information about the true causal relationships of a system and the causal effects of hypothetical interventions.</p><p>To achieve this goal, one has to follow a four-phased process.</p><ol><li>Come up with a causal graph of the system, either by expert knowledge or by performing causal discovery.</li><li>Use the causal graph to identify valid/optimal adjustment sets.</li><li>Use the adjustment sets to make interventions calculable from observational data, and pick a statistical learning method to fit your model to your data based on 1 &amp; 2.</li><li>Use your model to generate estimations of causal effects.</li></ol><p>To deal with possible nonlinearity, phase three is very crucial as it allows us to pick any kind of estimator to compute our causal quantities. This means that instead of Linear Regression (s. <a href="https://medium.com/causality-in-data-science/hands-on-causal-effect-estimation-with-python-aac40ca2cae0">blog post on basic causal effect estimation</a>), you could use KNN, Decision trees, or some fancy stuff like DNNs or XGBoost. This consequently makes the well-known trade-off between performance and interpretability as relevant to causal machine learning as it is to usual statistical learning. Nevertheless, the former benefits from the inherent trait of higher interpretability through being solely based on true causal relationships rather than spurious correlations and adding the opportunity to visualize causal effects across a range of interventions while also providing a graphical depiction of the system. This gets especially intriguing for non-linear and complex relationships for which usual linear regression is doomed to fail</p><p>So let’s start and explore this challenge of non-linearity in causal machine learning with a swing in our step, motivated by the notion that the tools of causal inference help us to master some of the most difficult problems in machine learning and beyond.</p><p>To perform causal effect estimation, we are going to use the Tigramite package. Being the go-to guy for full-stack causal inference with time-series (and non-time series) data, Tigramite provides continuously maintained and further developed state-of-the-art solutions with low package dependencies (Details: <a href="https://github.com/jakobrunge/tigramite">https://github.com/jakobrunge/tigramite</a>).</p><p>You can install it via the following command within the directory you want to use via the terminal:</p><pre>pip install tigramite</pre><p>Let’s continue and download the other packages for the tutorial:</p><pre># Imports<br>import numpy as np<br>import matplotlib<br>from matplotlib import pyplot as plt<br>%matplotlib inline     <br>## use `%matplotlib notebook` for interactive figures<br># plt.style.use(&#39;ggplot&#39;)<br>from scipy.stats import gaussian_kde<br>import tigramite<br>from tigramite import data_processing as pp<br>from tigramite.toymodels import structural_causal_processes as toys<br>from tigramite import plotting as tp<br>from tigramite.models import Models<br>from tigramite.causal_effects import CausalEffects<br><br>import sklearn<br>from sklearn.linear_model import LinearRegression<br>from sklearn.neighbors import KNeighborsRegressor<br>from sklearn.neural_network import MLPRegressor<br>from sklearn.gaussian_process import GaussianProcessRegressor</pre><p>As already indicated in the introduction, estimating and modeling nonlinear causal effects largely resembles the same process for linear causal effects. An example and a (very) quick introduction to the theory behind the processes can be found in our more elaborate blog post on <a href="https://medium.com/causality-in-data-science/hands-on-causal-effect-estimation-with-python-aac40ca2cae0">linear causal effect estimation</a>. Make sure to have a look at that one to understand what happens under the hood when performing causal effect estimation with Tigramite.</p><p>We now make a shortcut on this theoretical background, assuming it is already known and directly head into the Python tutorial.</p><h4>Generating Ground Truth Data</h4><p>Let’s start by first generating some non-linear ground truth data for both the general observations and the interventions needed to compute causal effects. To do so, we are using toys.structural_causal_process with Gaussian unit variance noise (noises = None) sampling data from the following SCM with a nonlinear function g:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3lgf3mman0LjMOZL_WjKjw.png" /></figure><pre>def lin_f(x): return x<br>def nonlin_f(x): return (x + 5. * x ** 2 * np.exp(-x ** 2 / 20.))<br><br>coeff = .5<br>links_coeffs = {<br>                0: [], <br>                1: [((0, 0), coeff, lin_f), ((5, 0), coeff, lin_f)], <br>                2: [((1, 0), -coeff, nonlin_f), ((5, 0), coeff, lin_f)],<br>                3: [((1, 0), 1., nonlin_f), ((2, 0), coeff, lin_f), ((6, 0),                    <br>                   coeff, lin_f), ((7, 0), coeff, lin_f)],<br>                4: [((5, 0), coeff, lin_f), ((7, 0), coeff, lin_f)], <br>                5: [],<br>                6: [],<br>                7: [],<br>                }<br><br># Observational data<br>T = 1000<br>data, nonstat = toys.structural_causal_process(links_coeffs, T=T, noises=None, seed=7)<br># Time series no 7 is unobserved confounder<br>data = data[:, [0,1,2,3,4,5,6]]<br>dataframe = pp.DataFrame(data)</pre><p>To estimate a causal effect of a joint intervention in X1 and X2 on Y, we could generate ground truth data for two interventions, i.e., x = 0, x’ = 1 as was done in our tutorial on causal effect estimation. However, since the causal dependencies are non-linear, we are interested in how the causal mechanism in our system behaves for different interventions. As a consequence, we generate interventional data at regular intervals on a range between -10 and 10 and calculate the true causal effect <em>E</em>[Y|do(X=x)) for setting each x-value.</p><p>The result is stored in true_causal_effects for later comparison with the estimates outputted by our causal model.</p><pre># Interventional data for a range of intervention values<br>intervention_data = np.linspace(-10, 10, 30)<br><br>true_causal_effects = np.zeros(len(intervention_data))<br>for i, int_val in enumerate(intervention_data): <br>    intervention1 = int_val*np.ones(T)<br>    intervention_data1, nonstat = toys.structural_causal_process(links_coeffs,<br>    T=T, noises=None, seed=7,intervention={0:intervention1, 1:intervention1}, <br>                                            intervention_type=&#39;hard&#39;,)<br>    # Time series no 7 is unobserved confounder<br>    intervention_data1 = intervention_data1[:, [0,1,2,3,4,5,6]]<br>    true_causal_effects[i] = intervention_data1[:,Y[0][0]].mean()</pre><p>Let’s make a quick pause here to ensure all is clear. We generated a total of 30 interventional values by manipulating the original system by the “intervention” argument in “structural_causal_process”. These data assets will then be used to calculate the true causal effect as a baseline. Also, we have generated one “observational” dataset from a non-intervened SCM on which we will train our causal model.<br>So in short: we have generated 31 data sets in total, 1 used to train the model and 30 to generate ground truth values for later performance assessment.</p><h4>Inference Preparation</h4><p>Now that we have generated our “observational” data, we can proceed and pursue our goal. That is, we want to build a causal model which is identifying the causal relationships and quantifying them for robust predictions and more causal effect estimations.<br>If we start from the beginning of the causal ML modeling cycle, we would have to first go through the causal discovery process, using methods like PCMCI+ to identify the causal graph representing the underlying system (for more details see our blog post on causal discovery).<br>For now, though, we will act as if the graph has already been identified or is known and solely focus on the part of modeling the system to estimate the causal effects.<br>To estimate causal effects in Tigramite we need to run the CausalEffects function for which we need to specify the following arguments:</p><blockquote>graph, graph_type, hidden_variables.</blockquote><p>graph is a string array of dtype=’&lt;U3&#39; in different shapes depending on graph_type:</p><ul><li>graph_type=’dag’ denotes a non-time series DAG where the graph is of shape (N, N) with N denoting the number of nodes.</li><li>graph_type=’admg’ denotes a non-time series ADMG (acyclic-directed mixed graph). This graph type can encode unobserved confounding using bidirected edges between nodes “&lt;-&gt;”, in addition to the standard directed edges “ →” For more details: see <a href="https://github.com/jakobrunge/tigramite/blob/master/tutorials/tigramite_tutorial_general_causal_effect_analysis.ipynb">GitHub</a>.</li><li>graph_type=’stationary_dag’ denotes a stationary time series DAG where the graph is of shape (N, N, tau_max + 1) where tau_max is the maximal time lag. The +1 stands for contemporaneous relations. (if the graph vocabulary is unfamiliar to you, have a look at our past blog posts on (time-series) DAGs)</li></ul><p>hidden_variables is more for special use cases and not discussed here. You can encode hidden variables using ADMGs, see above.</p><p>To construct a graph we provide all its edges in a NumPy array. Given we have identified the causal links, we can then proceed and specify an array that represents the graph underlying the SCM specified above. As already said, we assume that the graph is known such that we can specify the graph as follows:</p><pre>#graph in the toy model<br>graph =  np.array([[&#39;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;],<br>                   [&#39;&lt;--&#39;, &#39;&#39;, &#39;--&gt;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;&lt;--&#39;, &#39;&#39;],<br>                   [&#39;&#39;, &#39;&lt;--&#39;, &#39;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;&lt;--&#39;, &#39;&#39;],<br>                   [&#39;&#39;, &#39;&lt;--&#39;, &#39;&lt;--&#39;, &#39;&#39;, &#39;&lt;-&gt;&#39;, &#39;&#39;, &#39;&lt;--&#39;],<br>                   [&#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;&lt;-&gt;&#39;, &#39;&#39;, &#39;&lt;--&#39;, &#39;&#39;],<br>                   [&#39;&#39;, &#39;--&gt;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;&#39;],<br>                   [&#39;&#39;, &#39;&#39;, &#39;&#39;, &#39;--&gt;&#39;, &#39;&#39;, &#39;&#39;, &#39;&#39;]], dtype=&#39;&lt;U3&#39;)</pre><p>We can then plot the graph object:</p><pre>var_names = [&#39;$X_1$&#39;, &#39;$X_2$&#39;, &#39;$M$&#39;, &#39;$Y$&#39;, &#39;$Z_1$&#39;, &#39;$Z_2$&#39;, &#39;$Z_3$&#39;]<br>tp.plot_graph(graph = graph,<br>        var_names=var_names, <br>        save_name=&#39;Example.pdf&#39;,<br>        figsize = (8, 6),<br>        ); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYrvzC85N7l6Ikr0x9iuVg.png" /></figure><p>As you can see, there are directed edges, but also we see that there is an unobserved confounder assumed between Y and Z_1. With this being specified, we can then initialize a causal_effects object:</p><pre>#denotes the position in graph definition above: X_1 = first sub-array; X_2 = second sub-array<br>X = [(0,0), (1,0)] <br>Y = [(3,0)]<br><br>#Init CausalEffects<br>causal_effects = CausalEffects(graph, graph_type=&#39;admg&#39;, X=X, Y=Y, S=None, hidden_variables=None, verbosity=1)</pre><p>To obtain the optimal adjustment set for rendering the causal effects “identifiable” from observational data, we can then run the get_optimal_set() function.</p><pre>opt = causal_effects.get_optimal_set()<br>print(&quot;Oset = &quot;, [(var_names[v[0]], v[1]) for v in opt])</pre><p>Resulting in the output: <em>Oset = [(‘$Z_1$’, 0), (‘$Z_2$’, 0), (‘$Z_3$’, 0)]</em></p><p>So in this case, the optimal adjustment set consists of Z3 and Z2, being the parents of Y and M, and Z1 which is part of a collider path emanating from Y and M. Adjusting for these variables on the one hand blocks non-causal paths and on the other hand also yields favorable statistical properties (hence the name “optimal”, more <a href="https://arxiv.org/abs/2102.10324">here</a>).</p><h4>Causal Machine Learning Modeling</h4><p>This being done, we can move to estimate total causal effects utilizing graphical generalized backdoor adjustment with the optimal adjustment set. To do so, we use the average treatment effect estimation implemented in the fit_total_effect and predict_total_effect functions. Essentially, what we do is, take our causal_effect object with the specified optimal adjustment_set, pick an estimator (Tigramite uses sklearn estimators) and fit it to our dataframe following the logic of valid adjustment as defined in the equation above. We can thereby choose any algorithm that appears suitable. In this example, we choose a plain vanilla KNN to account for possible nonlinear dependencies.</p><pre>causal_effects.fit_total_effect(<br>        dataframe=dataframe, <br>        estimator=KNeighborsRegressor(),<br>        adjustment_set=&#39;optimal&#39;,<br>        conditional_estimator=None,  <br>        data_transform=None,<br>        mask_type=None,<br>        )</pre><p>And that’s all the magic. We have now trained a KNN-based causal machine learning model that is solely built upon the true identified causal dependencies.</p><h4>Performance Assessment</h4><p>In the last step, we now assess how close our model of the causal effect dynamics is to the actual ground truth generated from the SCM earlier. To do so, we can build a plot showing the true and estimated causal effects across all interventional values. For interpretation purposes of possible problems of this first model, it helps to also plot the density of X1.</p><pre>plt.scatter(intervention_data, estimated_causal_effects[:,0], label=&quot;Estimated&quot;)<br>plt.plot(intervention_data, true_causal_effects, &#39;k-&#39;, label=&quot;True&quot;)<br>plt.title(r&quot;NRMSE = %.2f&quot; % (np.abs(estimated_causal_effects[:,0] - true_causal_effects).mean()/true_causal_effects.std()))<br>plt.xlabel(&#39;Intervention / X-value range&#39;)<br>plt.ylabel(&#39;Causal effect&#39;)<br>plt.legend()<br><br># Also show observational density of X1<br>ax2 = plt.gca().twinx()<br>density = gaussian_kde(data[:,1])<br>ax2.plot(intervention_data, density(intervention_data), &#39;r-&#39;, label=r&quot;$p(X_1)$&quot;)<br>ax2.set_ylabel(r&#39;$p(X_1)$&#39;)<br>ax2.legend(loc=&#39;upper right&#39;)<br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-w1a6XxGyUTpknBvPM2FRw.png" /></figure><p>As can be seen, our model captures the nonlinearity of the causal effect quite well for an intervention range in which the density of X1 is not below zero. Outside this range, the KNN-estimator model cannot learn functional relations. Depending on whether this range is crucial in the application case or not, we could then proceed to a systematic process of model selection using well-known methods of grid search, trying out different hyperparameter values for the same model or different algorithms in general. However, you cannot overcome the usual statistical problem that one cannot learn how a function behaves where there is no training data if you don’t make strong functional constraints.</p><h4>Summary &amp; an Exercise</h4><p>Congratulations, you have just successfully created your model, for nonlinear causal relationships. This is something, as from now on you can avoid simply learning patterns from data without thinking about the relevant relationships underneath it. Instead, you know of a path to build robust models based on causal knowledge and a rigorous mathematical foundation. <br>We appreciate you accompanying us on this journey and hope that you will stick around and have a look at some of the state-of-the-art methods that we plan to publish on this platform in the future.<br>You are now ready to go on and explore the world of causal machine learning on your own.<br>As a small exercise, try to improve the causal model’s performance using your existing knowledge about training machine learning algorithms in the Sklearn library, and let us know what you come up with!<br>Happy modeling and thanks again for reading!</p><p><strong>About the authors:</strong></p><p><em>Kenneth Styppa was part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at QuantumBlack and is currently pursuing his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jakob Runge is Professor of AI in the Sciences at Potsdam University. His Causal Inference Lab develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.causalinferencelab.com/"><em>www.causalinferencelab.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4edfd8251a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/nonlinear-causal-effect-estimation-with-python-b4edfd8251a9">Nonlinear Causal Effect Estimation with Python</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validation in Causal Feature Learning]]></title>
            <link>https://medium.com/causality-in-data-science/validation-in-causal-feature-learning-5079f3c731dc?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/5079f3c731dc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 16:27:47 GMT</pubDate>
            <atom:updated>2025-01-16T16:27:47.735Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0bsRSpJ05bSxC5I0WEebQ.png" /></figure><h4>Causal Feature Learning Pt. 3</h4><p><em>In this article, we introduce the second stage of CFL on a toy example, which is Causal Validation. Before reading on make sure you have read Pt. 1 and Pt. 2 on Causal Feature Learning.</em></p><h4>The second stage of CFL: Causal validation</h4><p>In the previous stage of the CFL algorithm, we extracted a set of causal hypotheses for the causes and effects. More specifically, we have generated a set of possible causal macrostates for the source and the target variable. In this second stage, we will test if they are valid, i.e., correspond to true causal effects.</p><p>In an ideal scenario, we fulfill the assumptions necessary for applying the do calculus. In that case, we could use our observational data with the learned features, apply the do-calculus for intervention, and identify causal links.</p><p>Unfortunately, our toy example is more complicated than that, although it initially seems very simple. We work with a very simple causal graph with two nodes, where we know the cause and the effect. Additionally, we might have unobserved confounding. But that exactly poses the challenge, because we can’t be sure to have no confounding as it might just be unobserved. So to be on the safe route, we have to expect confounding to be present. Unfortunately, our causal graph, which now has three nodes, the cause, the effect, and the unobserved confounder, does not allow for do-calculus to estimate the true causal effects. Thus, the only remaining option left is to conduct experiments.</p><p>Therefore, we need to be able to intervene not only on the microstates of <em>hue</em>, but also on the macrostates that we have discovered. In some experimental setups, we may even be able to intervene only on the macrostates but not on the microstates. For example, consider particle kinetic energies as microstates and temperature as a macrostate. In this case, we cannot easily change the kinetic energy of a single particle, but we can change the average kinetic energy of all particles by adjusting the temperature. However, we should keep the ability to intervene in mind when choosing our source domain or when coming up with macrostates.</p><p>We conduct the experiments by setting the microstates of <em>hue’s</em> explicitly as we can control those states in our experiment. The authors of the CFL paper refer to this intervention on the microstate as <em>man</em> for “manipulation” to distinguish it from the intended <em>do</em>-intervention on the macrostate. To conduct a <em>do</em>-intervention on the macrostate, we set the experimental <em>hue </em>to any value of the microstates within a particular hypothetical causal macrostate of a cause. We can pick any value we like, as all <em>hue</em> values associated with the same macrostate are assumed to have the same causal effect under our model.</p><p>So in our experiment, we choose four <em>hue</em> levels. One for each source macrostate:</p><p>hue_experiments <strong>=</strong> [50., 140., 220., 340.];</p><p>Now we conduct our experiments, i.e., in our case, we repeat the simulation without the <em>hue’s</em> being dependent on <em>latitude </em>as this connection is cut by our intervention. This corresponds to removing the confounding by <em>latitude</em>. This leads to the following new causal graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PZaQbr5ijI3iHfOu" /></figure><p>Note that the confounding of latitude has been removed. Therefore, the distribution of <em>hue</em> no longer depends on latitude (<em>lat</em>). The conditional probability distributions for <em>eda</em> do not change, because the intervention keeps the causal mechanism intact. Nevertheless, the probabilities of reaching the different conditions, (R, lat), change, as there is no dependence between them anymore. As discussed before, we can now observe the true (well, we are still in a simulation) direct (unconfounded) causal effect of <em>hue</em> on <em>eda</em>.</p><p>The following code samples new data from our experiment under the change that <em>hue</em> no longer depends on the latitude (<em>lat</em>). This is our simulation equivalent of conducting the intervention experimentally.</p><pre>man_joint = map(hue_experiments) do hue_exp<br>    man_hue, eda_exp, _ = sample_eda_from_manipulation(hue_exp;<br>        nsamples=30)<br>    (man_hue, eda_exp)<br>end<br><br>man_hues = reduce(vcat, first.(man_joint))<br>eda_exps = reduce(vcat, last.(man_joint));</pre><p>We obtain the following simulated data (think experimental results of our intervention). You can see that we only obtain values for <em>eda</em> at our chosen values of <em>hue</em>.</p><pre>function manipulation_plot(man_hues, eda_exps)<br>    df = (hue=man_hues, eda=eda_exps)<br>    hueeda = data(df) * mapping(:hue, :eda) * visual(Scatter)<br>    draw(hueeda; axis=(<br>        xticks=[0., 90., 180., 270., 360.],<br>        yticks=[0., 0.5, 1.0],<br>        title=&quot;Experimental eda responses to manipulations&quot;<br>    ))<br>end<br><br>manipulation_plot(man_hues, eda_exps)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fb25cCXFSdXbAO_0gzW1lQ.png" /></figure><p>Next, we merge those possible causes that can lead to the same distribution of causal effects. As we have removed the spurious correlation from the unobserved confounding, we should be able to identify equal causes.</p><p>We express the causal effects as the conditional probabilities for <em>eda</em> under the manipulation of <em>hue</em>, p(eda|man(hue=hueexp)). Recall, that we call it manipulation because we had to intervene on the microstates, which are real physical states. Nevertheless, we can alternatively consider the macrostates, which are states that only exist conceptually, to obtain a simpler conditional probability for the macrostate of <em>eda</em>, E, given the macrostate of <em>hue</em>, H, p(E|do(H=Hexp)). We can estimate these conditional probabilities by counting the co-occurrences of the macrostates. Conditional probabilities essentially count how often one event might happen given another. In practice, we might have to consider too many possible events that manually counting them becomes impossible, which is why we work with suited probability distributions instead. Fortunately, in our example (and probably in many other problems where we might use CFL), we only have a few events to consider. These events correspond to our source and target macrostates, four each. So, to keep it easy, we just count the co-occurrences of those to estimate the conditional probabilities.</p><p>First, we have to assign the new experimental data to our discovered causal macrostates/hypotheses using the clustering we have fitted on the observational data.</p><pre>assigned_E = kmpredict(kmeda, grid_model_dens_for_eda(eda_exps))<br>assigned_H = kmpredict(kmhue, grid_model_dens_for_hue(man_hues));</pre><p>Next, we can estimate the conditional probabilities by counting co-occurrences of the source and target macrostates.</p><pre>function conditional_probabilities_from_counts(assigned_H, assigned_E)<br>    buckets = map(Iterators.product(sort(unique(assigned_H)),<br>        sort(unique(assigned_E)))) do (h, e)<br>            sum(assigned_E[assigned_H.==h] .== e)<br>    end<br>    buckets ./ sum(buckets; dims=2)<br>end<br><br>p_conds = conditional_probabilities_from_counts(assigned_H, assigned_E)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*znDwMx5ObNUswn9x2dQnyA.png" /></figure><p>Ignoring small differences, H=1(90,180) and H=3(180,270), as well as H=2(0,90) and H=4(270,360) have the same conditional probabilities for E=0 and E=1. Therefore, we merge those pairs of cause hypotheses and discover that <em>hue</em>’s ∈(90,270) account for one cause (Red=0) and <em>hue</em>’s ∈(0,90)∪(270,360) for the other cause (Red=1).</p><p>We can’t further merge the effect macrostates of <em>eda</em> as those are already associated with two distinct effects. Otherwise, we would have proceeded by combining those target macrostates with equal conditional probability distributions given the causes.</p><p>Usually, we will be able to merge macrostates when moving from the causal hypotheses to the experimentally verified causes and effects. A simple explanation is that confounding introduces additional dependencies between cause and effect, making it seem like a more complex mechanism.</p><p>Concluding, we have detected the true causal graph.</p><h4>Drawbacks of CFL</h4><p>As was apparent in this last article on CFL, the main drawback is that the method needs causal validation as an experimental step to identify the causal structure. Nevertheless, the grouping of multiple equivalent microstates into a few macrostates, providing the causal hypotheses, informs the experiments which should be conducted. The number of these experiments is often not that high and, therefore, practically feasible. Additionally, one can’t avoid experimentation to discover the causal structure unless one is willing to impose more assumptions. That said, CFL probably offers a good alternative to imposing assumptions that need to be verified by minimizing the amount of experimentation required. In general, however, we also argue that the learned features in the first stage are valuable for a variety of data science applications without being experimentally validated.</p><h4>CFL at a glance</h4><p>Alright, that was quite some material to chew on. Let’s try to pin down CFL into a short recipe that you can try to apply to your `projects:</p><ol><li>We need to know what is the source (cause) and target (effect) domain. The observations are low-level features, we refer to their values as microstates. We need to set the number of discrete source and target macrostates which will correspond to the high-level features that we try to extract from the data.</li><li>Estimate probability densities of our microstates, especially the conditional probability of the target microstates given the microstates.</li><li>Group together (cluster) those source microstates into the same macrostate that could lead to the same target microstates with equal probability. We did this with k-means clustering.</li><li>Group together (cluster) those target microstates into the same macrostate that could have resulted from the same source microstates with equal probability. We did this with k-means clustering.</li><li>Having discovered high-level features that explain the observations, we apply interventions on the source microstates to estimate the causal effects on the target microstates. We use the discovered source macrostates to guide our experiments by setting microstates to any value within the different macrostates.</li><li>Next, we observe the causal effects of the interventions. We assign the new observations, which are still low-level microstates, to the macrostates we identified before. The observed causal effects may lead us to merge some source macrostates if they have the same causal effect or to merge some target macrostates.</li><li>Finally, the remaining causal macrostates and the observed causal effects (describing the connections between macrostates) define the causal structure of our data.</li></ol><p>We hope that you have learned something interesting from our articles on CFL. One can conclude that it is a handy methodology for extracting causal insights from complex, seemingly overwhelming data, especially if one already knows that the data has to be simplified to become informative. Special thanks go to the authors of CFL (Chalupka et al., 2015a, 2015b, 2016, 2017) for proposing this method.</p><h4>Authors</h4><p><strong><em>Heiner Spieß </em></strong><em>is a PhD Student in the Neural Information Processing Group and Science of Intelligence Excellence Cluster at TU Berlin. His research focuses on understanding and improving the out-of-distribution generalization capabilities of Deep Neural Networks, e.g., through Causal Representation Learning. He is also interested in improving current statistical practice through causal inference and how Bayesian methods can be used to answer causal questions.</em></p><p><strong><em>Kenneth Styppa</em></strong><em> was part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at BMW and currently pursuing his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><strong><em>Jakob Runge </em></strong><em>is Professor of Data Science at TU Dresden. His Causal Inference Lab develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.causalinferencelab.com/"><em>www.causalinferencelab.com</em></a></p><h4>References</h4><p><em>Lin, D., Dai, J., Jia, J., He, K., Sun, J., 2016. ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Presented at the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Las Vegas, NV, USA, pp. 3159–3167. </em><a href="https://doi.org/10.1109/CVPR.2016.344"><em>https://doi.org/10.1109/CVPR.2016.344</em></a></p><p><em>Chalupka, K., Eberhardt, F., Perona, P., 2017. Causal feature learning: an overview. Behaviormetrika 44, 137–164. </em><a href="https://doi.org/10.1007/s41237-016-0008-2"><em>https://doi.org/10.1007/s41237-016-0008-2</em></a></p><p><em>Chalupka, K., Bischoff, T., Perona, P., Eberhardt, F., 2016. Unsupervised Discovery of El Nino Using Causal Feature Learning on Microlevel Climate Data.</em></p><p><em>Chalupka, K., Perona, P., Eberhardt, F., 2015a. Visual Causal Feature Learning.</em></p><p><em>Chalupka, K., Perona, P., Eberhardt, F., 2015b. Multi-Level Cause-Effect Systems.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5079f3c731dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/validation-in-causal-feature-learning-5079f3c731dc">Validation in Causal Feature Learning</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on Causal Discovery with Python]]></title>
            <link>https://medium.com/causality-in-data-science/hands-on-causal-discovery-with-python-e4fb2488c543?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4fb2488c543</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Mon, 04 Mar 2024 16:29:39 GMT</pubDate>
            <atom:updated>2024-03-10T14:09:51.481Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7W1tYzyzRdQtfYX81uDIg.png" /></figure><p>A Gentle Guide to Causal Inference with Machine Learning Pt. 9</p><p><em>You probably read all kinds of articles explaining the fundamentals of causal inference and its connection to machine learning by now. “This might sound nice”, you could think, “but how can I use it?!”. This is a valid and important question. A question that will be answered in this blog post.</em></p><p><em>More precisely, you will learn how you can use the PCMCI algorithm to identify causal graphs out of high-dimensional time-series data. An indeed difficult task. Difficult, but not impossible. Especially with the right packages to use. So let’s do it!</em></p><p><em>Before you keep on reading, please make sure you are familiar with the following concepts:</em></p><ol><li><a href="https://medium.com/causality-in-data-science/assumptions-for-causal-discovery-cc194d607a14"><em>Causal Markov, Faithfulness and Sufficiency Assumption</em></a></li><li><a href="https://medium.com/causality-in-data-science/confounding-colliding-d-separation-and-sleeping-with-shoes-on-8ba43c976354"><em>Fundamental idea behind Causal Inference</em></a></li><li><a href="https://medium.com/causality-in-data-science/why-machine-learning-needs-causality-3d33e512cd37"><em>The reason why Machine Learning needs Causality</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a"><em>Causal Graphs &amp; D-separation</em></a></li><li><a href="https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c"><em>The PCMCI algorithm</em></a></li></ol><h4>Getting started:</h4><p>Alright, we will get our hands dirty now. Finally, you will learn how to do algorithmic causal inference.</p><p>To do so, we are going to use the Tigramite package. Being the go-to guy for full-stack causal inference with time-series data, Tigramite provides continuously maintained and further developed state-of-the-art solutions with low package dependencies (<a href="https://github.com/jakobrunge/tigramite">GitHub repository</a>).</p><p>You can install it via pip. <strong>Make sure the Python version in your environment matches the requirements which you will find </strong><a href="https://github.com/jakobrunge/tigramite/tree/master?tab=readme-ov-file#required-python-packages"><strong>here</strong></a><strong> updated regularly</strong>.</p><pre>pip install tigramite</pre><p>This being done, you can now proceed to import all the packages you need for your analysis (the list below may be more than you need):</p><pre># Imports<br>import numpy as np<br>import matplotlib<br>from matplotlib import pyplot as plt<br>%matplotlib inline     <br>## use `%matplotlib notebook` for interactive figures<br># plt.style.use(&#39;ggplot&#39;)<br>import sklearn<br><br>import tigramite<br>from tigramite import data_processing as pp<br>from tigramite.toymodels import structural_causal_processes as toys<br><br>from tigramite import plotting as tp<br>from tigramite.pcmci import PCMCI<br>from tigramite.lpcmci import LPCMCI<br><br>from tigramite.independence_tests.parcorr import ParCorr<br>from tigramite.independence_tests.robust_parcorr import RobustParCorr<br>from tigramite.independence_tests.parcorr_wls import ParCorrWLS <br>from tigramite.independence_tests.gpdc import GPDC<br>from tigramite.independence_tests.cmiknn import CMIknn<br>from tigramite.independence_tests.cmisymb import CMIsymb<br>from tigramite.independence_tests.gsquared import Gsquared<br>from tigramite.independence_tests.regressionCI import RegressionCI</pre><p>One last remark before we start. We will have to work with a couple of indices here. This can be messy — especially in Medium. If we write X(j)_(t-tau) we mean the value of the i-th variable in our system, tau timesteps in the past from the present timestamp t. Or simply:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pc3rNSpDxHTaSM4zOWKb_A.png" /></figure><p>Alright, now we can start and do some fun causal discovery!</p><h4>What we want to model:</h4><p>Causal systems can be represented by SCMs (see <a href="https://medium.com/causality-in-data-science/structural-causal-models-a-quick-introduction-1ab49259e921"><em>here</em></a>). Of course, in reality, you would not have perfect information about the SCM that underlies your data. Instead, we will only have a bunch of observations of - if you assume causal sufficiency - all relevant variables of the underlying system. Meaning, that if your system can be sufficiently described by 4 variables, you have data about each one of them. Causal discovery should now help you to bring some light into this black box and identify causal relationships.</p><p>We will do this with an example now. In this example, contrary to reality, we already know from which SCM the data is generated, making it possible for us to check if the results we obtain from using PCMCI are correct. This is how our SCM with <em>η(j)_(t)</em> as independent zero-mean unit variance random variables looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RpQUSTNgUgi72Rd0-PsSA.png" /></figure><p>Hopefully, your data is a “random” sample of data points from the distribution over the SCM. We will now do the same and generate such data from our SCM. This means we randomly pick a value for <em>η(j)_(t) </em>for each of the <em>j</em> variables and each time step <em>t</em>, and as a result obtain values for each variable. As we observe a time-dependent system, each of those samples stands for one point in time, with the next point always “building upon” the previous one. We do this a thousand times in the code below to generate a time series of 4 variables with a length of 1000 using Tigramites toys.structural_causal_process function.</p><pre>np.random.seed(42)     # Fix random seed to make results reproducable<br>def lin_f(x): return x # A linear dependency function<br>links_coeffs = {0: [((0, -1), 0.7, lin_f), ((1, -1), -0.8, lin_f)],<br>                1: [((1, -1), 0.8, lin_f), ((3, -1), 0.8, lin_f)],<br>                2: [((2, -1), 0.5, lin_f), ((1, -2), 0.5, lin_f), ((3, -3), 0.6, lin_f)],<br>                3: [((3, -1), 0.4, <br>                } #stores the functional dependencies of the SCM<br>T = 1000     # time series length<br>data, true_parents_neighbors = toys.structural_causal_process(links_coeffs, T=T) #generate the timeseries<br>T, N = data.shape<br><br># Initialize dataframe object, specify time axis and variable names<br>var_names = [r&#39;$X^0$&#39;, r&#39;$X^1$&#39;, r&#39;$X^2$&#39;, r&#39;$X^3$&#39;]<br>dataframe = pp.DataFrame(data, <br>                         datatime = {0:np.arange(len(data))}, <br>                         var_names=var_names)</pre><p>Alright, now that we have generated our data let’s take a look at it with the tp.plot_timeseries function.</p><pre>tp.plot_timeseries(dataframe); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/0*3PY98oBQFmx2u3vF" /></figure><p>Yes, of course, the data is quite perfect. Usually, in real-time-series data, you will have problems such as non-stationarity, missing data or different time granularities. We won’t deal with these problems now and just focus on the basics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k1EVdRSWCMWQTFiD" /></figure><p>Alright so now the data is quite clear. The challenge at hand is to use PCMCI to detect the correct causal links — which requires that its underlying assumptions are fulfilled, see <a href="https://medium.com/causality-in-data-science/assumptions-for-causal-discovery-cc194d607a14"><em>Causal Markov, Faithfulness and Sufficiency Assumption</em></a>. PCMCI also requires no instantaneous causal links. Before getting to that let’s quickly understand on a high level what PCMCI does.</p><h4>A quick recap about PCMCI:</h4><p>Causal Discovery with PCMCI can be split into a process with two phases.</p><p><strong>Identifying a superset of the parents for all variables X(j) in the system using a variant of the PC algorithm. </strong>As outlined in <a href="https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c">another blog post</a>, PC is a flexible discovery algorithm, giving us an estimated set of parents per node that can account for nonlinear functional dependencies as well as continuous and discrete variables, depending on the chosen conditional independence test. Phase 1 is a kind of pre-selection step, that will help make the second step more efficient.</p><p><strong>Using MCI (Momentary Conditional Independence) to iteratively test conditional independencies. </strong>Given our background knowledge about the Causal Markov condition, we know that the causal parents of a variable are a sufficient conditioning set when testing for causal relationships as conditional dependencies because here no instantaneous causal links are assumed. Consequently, we can now use the parents identified in phase one, and use MCI to test whether “X(i)_(t-tau) -&gt; X(j)_t” with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*y7lLzd9KfqhwAzU8" /></figure><h4>Initializing &amp; Running PCMCI</h4><p>When using PCMCI we have to make a couple of decisions that you probably spotted in the quick recap above or the blog post on PCMCI.</p><p>First, we need to choose a conditional independence test for phase 1 and phase 2. Assuming our cause and effects to be linearly dependent we will use the partial correlation test specified with significance=’analytic’ that assumes the null distribution to be Student’s t and then initialize PCMCI using the defined independence test.</p><pre>parcorr = ParCorr(significance=&#39;analytic&#39;)<br>pcmci = PCMCI(<br>    dataframe=dataframe, <br>    cond_ind_test=parcorr,<br>    verbosity=1)</pre><p>The next decision we need to make is to pick the maximum time-lag tau_max on which we expect the longest possible causal link.</p><p>Without some theoretical a priori knowledge this decision might be quite hard. What helps here is to plot the lagged correlations of the time series to see at what lag no correlations exist anymore, making those lags irrelevant. This can be done using the following command (where tau_max is chosen to be very large to see a big timeframe):</p><pre>correlations = pcmci.get_lagged_dependencies(tau_max=20, val_only=True)[&#39;val_matrix&#39;]<br>lag_func_matrix = tp.plot_lagfuncs(val_matrix=correlations, <br>                                   setup_args={&#39;var_names&#39;:var_names, <br>                                   &#39;x_base&#39;:5, &#39;y_base&#39;:.5}); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/0*r66t6-JTNp-5SERR" /></figure><p>What we can see from the plots is that the correlations decay at a lag of around 8. Consequently, we pick tau_max=8. If you assume nonlinearities, you would have to use mutual information or a similar measure for this step.</p><p>That’s almost it. We now decide what kind of independence test we want to use on what kind of time range. The last thing to determine significant links now is to define when we perceive a dependence to be strong enough such that the null hypothesis can be rejected in the Phase 1 selection step. This significance value is chosen using the parameter pc_alpha, with a default value of 0.05, setting the significance level in the condition-selection step. Instead of just picking one value at random, we can also let PCMCI choose the optimal value by setting it to pc_alpha=None. Then PCMCI will optimize this parameter in the ParCorr case by the Akaike Information criterion among a reasonable default list of values (e.g., pc_alpha = [0.05, 0.1, 0.2, 0.3, 0.4, 0.5]). The last remaining parameter we need to specify now is alpha_level=0.01 which indicates that we threshold the resulting p-value matrix in the Phase 2 step at this significance level to obtain the graph.</p><p>With this, we are done and can run PCMCI to do the causal discovery for us. You can now run the algorithm and watch what happens:</p><pre>pcmci.verbosity = 1<br>results = pcmci.run_pcmci(tau_max=8, pc_alpha=None, alpha_level=0.01)</pre><h4>Interpreting the Output</h4><p>The output you get is quite long, so let’s make sure we understand what we see.</p><p>First, you see an output related to the PC phase. This includes first the parameters you have chosen:</p><pre>##<br>## Step 1: PC1 algorithm with lagged conditions<br>##<br>Parameters:<br>independence test = par_corr<br>tau_min = 1<br>tau_max = 8<br>pc_alpha = [0.05, 0.1, 0.2, 0.3, 0.4, 0.5]<br>max_conds_dim = None<br>max_combinations = 1</pre><p>Next, you see the outcomes of the PC1 algorithm. This includes the discovered parents for each variable, sorted by their p-value.</p><pre>## Resulting lagged parent (super)sets:<br> Variable $X⁰$ has 7 link(s):<br> [pc_alpha = 0.3]<br> ($X⁰$ -1): max_pval = 0.00000, min_val = 0.796<br> ($X¹$ -1): max_pval = 0.00000, min_val = -0.748<br> ($X³$ -2): max_pval = 0.13715, min_val = 0.048<br> ($X³$ -1): max_pval = 0.19326, min_val = -0.042<br> ($X²$ -1): max_pval = 0.20477, min_val = -0.041<br> ($X³$ -7): max_pval = 0.25822, min_val = 0.036<br> ($X¹$ -4): max_pval = 0.26548, min_val = 0.036<br> Variable $X¹$ has 4 link(s):<br> [pc_alpha = 0.3]<br> ($X¹$ -1): max_pval = 0.00000, min_val = 0.695<br> ($X³$ -1): max_pval = 0.00000, min_val = 0.484<br> ($X³$ -7): max_pval = 0.14660, min_val = 0.046<br> ($X²$ -1): max_pval = 0.22144, min_val = -0.039<br> Variable $X²$ has 4 link(s):<br> [pc_alpha = 0.2]<br> ($X²$ -1): max_pval = 0.00000, min_val = 0.450<br> ($X¹$ -2): max_pval = 0.00000, min_val = 0.447<br> ($X³$ -3): max_pval = 0.00000, min_val = 0.183<br> ($X³$ -1): max_pval = 0.12516, min_val = 0.049<br> Variable $X³$ has 1 link(s):<br> [pc_alpha = 0.05]<br> ($X³$ -1): max_pval = 0.00000, min_val = 0.372</pre><p>Of course, the parents above are not the final result of PCMCI. They are rather the set we will condition on in the second phase. The second big part of the output shows you the result of the MCI process. The output is again structured similarly to the PC-related output. First, you see the parameters and then the discovered links of each variable sorted by its p-value and described by the latter together with the test statistic value (here partial correlations).</p><pre>##<br>## Step 2: MCI algorithm<br>##<br><br>Parameters:<br><br>independence test = par_corr<br>tau_min = 0<br>tau_max = 8<br>max_conds_py = None<br>max_conds_px = None<br><br>## Significant links at alpha = 0.01:<br><br>    Variable $X^0$ has 3 link(s):<br>        ($X^1$ -1): pval = 0.00000 | val = -0.653<br>        ($X^0$ -1): pval = 0.00000 | val =  0.566<br>        ($X^1$ -5): pval = 0.00496 | val = -0.090<br><br>    Variable $X^1$ has 2 link(s):<br>        ($X^3$ -1): pval = 0.00000 | val =  0.663<br>        ($X^1$ -1): pval = 0.00000 | val =  0.622<br><br>    Variable $X^2$ has 3 link(s):<br>        ($X^3$ -3): pval = 0.00000 | val =  0.451<br>        ($X^1$ -2): pval = 0.00000 | val =  0.446<br>        ($X^2$ -1): pval = 0.00000 | val =  0.425<br><br>    Variable $X^3$ has 1 link(s):<br>        ($X^3$ -1): pval = 0.00000 | val =  0.372</pre><p>Alright, let&#39;s see the kind of links we discovered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0AoHadDvdKzDutqYboD9Zg.png" /></figure><p>If we compare our learned graph with the true SCM, we only discovered one erroneous link namely the one at X(1)_(t-5).</p><p>If we are a little more strict and set our significance level to 1%, we have no False Positives left and discover the correct graph. Try it yourself as an exercise!</p><p>Once you ran PCMCI and stored the results, you can always re-access them by printing out the p_matrix and val_matrix which are of the size (N, N, tau_max+1) with the entry (i, j, \tau) denoting the test for the link between X(i)_(t-tau) and X(j)_t.</p><pre>print(&quot;p-values&quot;)<br>print (results[&#39;p_matrix&#39;].round(3))<br>print(&quot;MCI partial correlations&quot;)<br>print (results[&#39;val_matrix&#39;].round(2))</pre><pre>p-values<br>[[[1.    0.    0.262 0.894 0.886 0.034 0.417 0.255 0.877]<br>  [0.067 0.296 0.095 0.765 0.502 0.242 0.234 0.899 0.951]<br>  [0.832 0.657 0.453 0.011 0.359 0.48  0.901 0.243 0.375]<br>  [0.827 0.656 0.495 0.592 0.643 0.976 0.775 0.931 0.759]]<br><br> [[0.067 0.    0.15  0.51  0.041 0.005 0.559 0.533 0.392]<br>  [1.    0.    0.911 0.07  0.576 0.556 0.175 0.732 0.741]<br>  [0.651 0.764 0.    0.231 0.495 0.43  0.423 0.944 0.238]<br>  [0.153 0.171 0.088 0.989 0.965 0.859 0.915 0.43  0.671]]<br><br> [[0.832 0.1   0.182 0.443 0.37  0.265 0.998 0.89  0.65 ]<br>  [0.651 0.072 0.594 0.387 0.883 0.765 0.696 0.44  0.69 ]<br>  [1.    0.    0.074 0.942 0.956 0.626 0.818 0.526 0.942]<br>  [0.788 0.575 0.332 0.658 0.603 0.65  0.816 0.055 0.851]]<br><br> [[0.827 0.03  0.021 0.597 0.635 0.2   0.392 0.081 0.85 ]<br>  [0.153 0.    0.558 0.534 0.72  0.284 0.798 0.016 0.162]<br>  [0.788 0.114 0.781 0.    0.975 0.098 0.265 0.161 0.642]<br>  [1.    0.    0.937 0.85  0.503 0.657 0.394 0.795 0.567]]]<br>MCI partial correlations<br>[[[ 0.    0.57  0.04  0.   -0.    0.07  0.03 -0.04 -0.  ]<br>  [-0.06 -0.03  0.05  0.01  0.02 -0.04  0.04  0.   -0.  ]<br>  [ 0.01 -0.01  0.02  0.08 -0.03 -0.02 -0.   -0.04  0.03]<br>  [ 0.01  0.01  0.02  0.02  0.01 -0.   -0.01 -0.   -0.01]]<br><br> [[-0.06 -0.65  0.05  0.02  0.07 -0.09  0.02  0.02  0.03]<br>  [ 0.    0.62  0.   -0.06 -0.02 -0.02  0.04 -0.01  0.01]<br>  [ 0.01 -0.01  0.45  0.04  0.02  0.03 -0.03 -0.   -0.04]<br>  [-0.05 -0.04 -0.05  0.   -0.   -0.01 -0.   -0.03  0.01]]<br><br> [[ 0.01 -0.05 -0.04  0.02  0.03  0.04 -0.    0.   -0.01]<br>  [ 0.01  0.06  0.02 -0.03 -0.   -0.01 -0.01 -0.02 -0.01]<br>  [ 0.    0.43 -0.06  0.   -0.    0.02  0.01  0.02  0.  ]<br>  [-0.01 -0.02  0.03 -0.01 -0.02 -0.01  0.01  0.06  0.01]]<br><br> [[ 0.01 -0.07  0.07 -0.02  0.02  0.04 -0.03  0.06 -0.01]<br>  [-0.05  0.66  0.02 -0.02 -0.01  0.03  0.01  0.08 -0.04]<br>  [-0.01  0.05 -0.01  0.45 -0.   -0.05  0.04  0.04 -0.01]<br>  [ 0.    0.37 -0.   -0.01 -0.02 -0.01  0.03 -0.01  0.02]]]</pre><p>If you look at the test statistic value (val) right now, you might wonder whether it serves as a good indicator of the strength of the causal effect. And your intuition is not misleading, it can indeed be a good indicator. Yet, it does not replace a sound causal effect estimation procedure, which we will discuss in the next article of this series.</p><h4>Give it some colour</h4><p>Great, now you have a mess of numbers. Let’s visualize it to make it easier for people to see what kind of message you want to bring across.</p><p>To do so Tigramite offers two kinds of plots both being created with very similar functions that take the graph array of the results, the val_matrix and the var_names as inputs. More precisely, you can choose between a time series causal graph or a process causal graph that summarizes the first one. Both can be very helpful depending on what kind of audience you have and if the time-dependency of the process needs to be depicted or not.</p><p>If for example, you want to show the causal system you have discovered with respect to its time-dependency, you can plot a time-series graph similar to those that we have discussed in the fundamental blog post about PCMCI:</p><pre># Plot time series graph    <br>tp.plot_time_series_graph(<br>    figsize=(6, 4),<br>    val_matrix=results[&#39;val_matrix&#39;],<br>    graph=results[&#39;graph&#39;],<br>    var_names=var_names,<br>    link_colorbar_label=&#39;MCI&#39;,<br>    ); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*GyjzTwUqrjhcQpX55fWKEQ.png" /></figure><p>As you see above, the ts-graph you obtain will show all the lags for the tau_max you selected. It shows you all the nodes representing the variables at different points in time, with the links between them visualizing the cross-MCI value.</p><p>While the ts-graph is undoubtedly more precise and informative regarding the time dependencies between the variables, it is also quite hard to grasp at first glance. This makes it a little disadvantageous when exposing the plot to an audience that is not very knowledgeable in the field of causal inference.</p><p>The so-called process graph could help here. It allows you to give a simple overview of your results, with a plot that summarizes the information of a ts-graph into something less crowded. This is achieved by “shaving off” the time-dependencies in the plot:</p><pre>tp.plot_graph(<br>    val_matrix=results[&#39;val_matrix&#39;],<br>    graph=results[&#39;graph&#39;],<br>    var_names=var_names,<br>    link_colorbar_label=&#39;cross-MCI&#39;,<br>    node_colorbar_label=&#39;auto-MCI&#39;,<br>    ); plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*tjFi4KltTJYZT9qEzoi-Xg.png" /></figure><p>In such process graphs the node color denotes the auto-MCI value and the link colors the cross-MCI value. If links occur at multiple lags between two variables, the link colour denotes the strongest one and the label lists all significant lags in order of their strength, minimizing the information loss although the time dimension is more or less ignored.</p><h4>Wrapping it up</h4><p>Congratulations! You have just done your first successful causal discovery in Python and at the same time found a way to analyze and visualize your findings intuitively.</p><p>Please be aware though, that your causal discovery is only as good as the data you use and whether it justifies the assumptions you make. If for example, you cannot assume causal sufficiency, interpreting any detected link as existing can be optimistic. Have a look at <a href="https://github.com/jakobrunge/tigramite/blob/master/tutorials/Runge_Causal_Inference_for_Time_Series_NREE.pdf">this guide/review paper</a> for a more in-depth tour of causal discovery.</p><p>All in all, the kind of interpretation you can make largely depends on the application scenario. While we have mentioned certain stratagems that all point toward scientific scepticism, there is no real recipe that works all the time. Rather you have to make informed choices based on the data at hand.</p><p>Addressing this challenge, Tigramite offers a set of additional algorithms and independence tests that could help you in your specific projects. <strong>Tutorials about these tests, algorithms, and caveat stratagems can be found on </strong><a href="https://github.com/jakobrunge/tigramite"><strong>Tigramite’s GitHub Page</strong></a><strong>.</strong></p><p>But wait, that’s not it. Causal Inference is about more than discovering causal links. It is also about estimating the size of the effects right? So once you have discovered a causal graph you are not at the end of the road. Well, we hope you keep reading because this part of the journey is material for another blog post.</p><p>Thanks!</p><p><strong>About the authors:</strong></p><p><em>Kenneth Styppa is part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist in Companies such as BMW and QuantumBlack and currently pursues his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jakob Runge heads the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em> in Jena and is chair of computer science at TU Berlin. The Causal Inference group develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.climateinformaticslab.com/"><em>www.climateinformaticslab.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4fb2488c543" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/hands-on-causal-discovery-with-python-e4fb2488c543">Hands-on Causal Discovery with Python</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structural Causal Models — A Quick Introduction]]></title>
            <link>https://medium.com/causality-in-data-science/structural-causal-models-a-quick-introduction-1ab49259e921?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ab49259e921</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 10:04:07 GMT</pubDate>
            <atom:updated>2023-08-29T10:04:07.843Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qMfLgrEAIbGkbNN3ISCrnQ.png" /></figure><h3>Structural Causal Models — A Quick Introduction</h3><h4>A Gentle Guide to Causal Inference with Machine Learning Pt. 7</h4><p><em>From reading through the past blog posts, you are familiar with the basic idea of causal inference and how we use certain assumptions and methods such as conditional independence tests to identify a causal graph that describes the causal relations among our observed variables. While this is undoubtedly already quite an achievement, the world of causal inference has even more to offer. Let’s say there would be a way to formalize the whole system into a set of neat equations that enables you to assess all kinds of intriguing things such as quantifying the effect of interventions or even reason about counterfactuals… Wouldn’t that be amazing? Well, it is exactly what Structural Causal Models (SCM) can offer.</em></p><p><em>You should be familiar with the following concepts from the previous articles before you keep on reading:</em></p><ol><li><a href="https://medium.com/causality-in-data-science/assumptions-for-causal-discovery-cc194d607a14"><em>Causal Markov, Faithfulness, and Sufficiency Assumption</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-is-causal-machine-learning-ceb480fd2902"><em>Fundamental idea behind causal inference</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a"><em>Causal Graphs</em></a><em> &amp; </em><a href="https://medium.com/causality-in-data-science/confounding-colliding-d-separation-and-sleeping-with-shoes-on-8ba43c976354"><em>D-separation</em></a></li><li><a href="https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c"><em>Causal Discovery &amp; Conditional Independence Tests</em></a></li></ol><h4>The idea behind SCMs</h4><p>Before getting to the formalities, let’s understand the very basic idea behind SCMs.</p><p>So far, the only type of causal models that we have encountered are Causal Graphs, directed acyclic graphs that depict causal relations in a binary way. Either there is a cause-effect relation between two variables, in which case there is a directed edge from cause to effect in the graph, or there is none, and therefore there is also no edge.</p><p>Ideally, we would like to expand this binary description about causal relations to a more fine-granular one that includes more details. For example, being able to quantify the effect of certain policy measures on climate change would be more helpful than “just” knowing that they are causally related. Especially, when there would be several policies to choose from. But how could we get this done?</p><p>In a way, we just need to add some additional information to a Causal Graph, describing the characteristics of the discovered causal relationships.</p><p>In other words, we formulate a set of equations that describe all causal relations in our system. Let’s say we have discovered the following graph using the methods described in our earlier blog posts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DBorg8KY0L_26rzvUvxl6g.png" /></figure><p>In an SCM, all four variables will have their own equation, describing the functional mechanism that explains how a variable’s parents influence the variable itself. But SCMs include not only the influence of the parents, because we want the causal relations to be described in a probabilistic manner. Meaning, instead of saying that, every time when X changes by 1 unit, Y changes by 5 units, we acknowledge that there will be some random noise (N) that could alter the actual effect one way or another. This means that our SCM for the example graph above looks as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFWKJmACu4So8sZehzYApA.png" /></figure><p>A formal definition is provided in the book Elements of Causal Inference from Peters et al. (2017):</p><h4>Definition of Structural Causal Models</h4><p>We give the definition as provided in the Elements of Causal Inference from Peters et al. (2017):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1iHWZ6ui5Ml92avsv_V-w.png" /></figure><h4>Facets of Structural Causal Models</h4><p><strong>The world is uncertain — Random Noise in SCMs</strong></p><p>As mentioned before, SCMs are probabilistic due to the random noise terms that appear in the equations. Additionally, for the rest of this post, we make an important assumption: the noise terms must be independent. In a way, we assume that our model is able to explain all systematic dependencies in our system and there is no unobserved confounding left. Be careful though: if you do think that some of your variables are confounded by something that you haven’t measured, you will have to work with SCMs of a more general kind — or you include another, unobserved node in your graph.</p><p><strong>Keep it simple — Structural Minimality</strong></p><p>The art of explaining is doing so in the simplest possible way. This principle has found its way into science under the name of Occam’s Razor, and states that among equally good explanations, the simplest one is to be preferred.</p><p>An incarnation of Occam’s Razor in Causal Inference goes by the name of Structural Minimality. It requires that all functions f_j describe an actually non-zero dependency between X_j and the function’s input arguments. This implies that the following SCM</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n7lb2DoCr7crucdXzYZ1nQ.png" /></figure><p>has to be rewritten into:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ya-gyIobola4lDDQT_j_ww.png" /></figure><p>in order to satisfy Structural Minimality. If we derive a Causal Graph from an SCM by drawing arrows from a function’s input to its output variables, Structural Minimality ensures that each edge we draw actually corresponds to non-zero causal effects.</p><p><strong>The independence of cause and mechanism in SCMs</strong></p><p>In an earlier blogpost we introduced the mechanism as the facilitator that brings about the effect from its cause. In this sense, the function f_j is the mechanism that connects X_j to all its parents. In other words, f_j is the mechanism by which X_j is caused.</p><p>Also, we have already explained that we assume these mechanisms to be invariant of shifts in the cause or in other mechanisms. That is, no matter what we do to the value of the parent variable or to the other mechanisms, the mechanism f_j, by which X_j is brought about, stays the same. This also has important implications for the next question we ask ourselves.</p><h4>How to learn Structural Causal Models</h4><p>Say that we have already found the Causal Graph describing our system, for instance through a previous causal discovery process, and we would like to learn the more fine-grained SCM. We will illustrate how to do this using the following graph, corresponding to our example SCM above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*gwOQgymiAJhuWxUsMoq29w.png" /></figure><p>Having already identified the existence of causal relations in step one, we now try to quantify them through functions that resemble the observable variability of the data as closely as possible. So we know that X_1 depends on X_3, but now we want to identify the structural assignment X_1:=f_1(X_3, N_1).</p><p>The wonderful part about causal inference is that if we have done our homework in the previous steps, this last identification / quantification step can be done with all the tools we know from classical statistical / machine learning. For example, we could learn f_1 using OLS, decision trees, ANNs and so forth with the correct variables.</p><p>Of course, we cannot arbitrarily choose any kind of model and any variables we fancy to be interesting. Again, we need to check what kind of assumptions we can make (e.g., such as linearity for OLS), back it up with some theory and run the adequate models to learn and quantify the previously detected dependencies. For instance, let’s say you have detected a dependency of X_1 on X_3 in a previous causal discovery process. In addition, you have reasons to assume the link to be highly non-linear. You can then train a Neural Net to learn the non-linear function as you have done many times before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6-HZlU3no82bSl721xUhZA.png" /></figure><h4>SCMs and Interventions</h4><p>Interventional questions are all around us. Questions such as “what would happen if a climate policy X would come into effect?”. These “what would happen with Y if X is set to a certain value” type of questions are typical for causal inference. The kind of actions they describe are so-called interventions. In the end, they describe the process of intervening on a system by setting one or more variables of a system to take on a specific value, as one does in a scientific experiment.</p><p>This is clearly different from conditioning, where we select those observations out of a sample that takes on a certain value while ignoring the others. Intervening is not selecting, it is forcing all instances to take on this value.</p><p>Consequently, the term P(Y|do(X = 2)) , which expresses the probability distribution of Y if X is set to 2, does not generally equal the conditional probability P(Y|X = 2). For a deeper explanation of this statement, have a look at our past blog posts.</p><p>Precisely because conditioning and intervening is not the same, questions such as the one about climate policy can’t be answered simply by comparing two means.</p><p>SCMs come in very handy here. Assume that we have conducted proper causal discovery (or we just know the qualitative graph by expert knowledge) and SCM estimation, we can now model intervention distributions quite easily. Let’s again understand this using the 4 variable example from above. The SCM still looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*haH0VyXwq03VuQDkbAE56Q.png" /></figure><p>The effect of a hard intervention do(X_1=2) can be modeled by the following alteration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SvlFqUAG0ga0sggw0GwBEw.png" /></figure><p>We simply set the variable on which we perform the intervention to our value of choice. The effect of changing X_1 will then “ripple” through the whole system and will change the outcome of all its descendants, in this case X_2 and X_4.</p><p>This new SCM entails a new distribution, the interventional distribution.</p><p>If you recall one of our first articles, this is one of the great benefits of causal “infused” machine learning. We can make these adaptations and still have a solid model even, or especially, when facing severe distribution shifts.</p><p>More details on learning SCMs and the challenges that might arise such as not knowing the causal variables in the first place can be found in: <a href="https://arxiv.org/pdf/2210.13583.pdf">https://arxiv.org/pdf/2210.13583.pdf</a>. And also have a look at our recently published Review/Howto-guide article <a href="https://t.co/wzp7UvcvXX">published in Nature Reviews Earth and Environment.</a></p><p>With this blogpost we now covered the basic fundamentals and will keep on exploring more advanced topics. Keep on reading to earn the fruits of all the fundamentals we built up! Up next — multivariate causal discovery in time series settings.</p><p><strong>Source and Reading Recommendation:</strong></p><p><em>Peters, J., Janzing, D., &amp; Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms (p. 288). The MIT Press.<br>Runge, J., Gerhardus, A., Varando, G., Eyring, V., Camps-Valls, G. (2023). Causal Inference for Time Series. Nature Reviews Earth and Environment 4, 487–505. View-only version </em><a href="https://t.co/wzp7UvcvXX"><em>here</em></a><em>.</em></p><p><strong>About the authors:</strong></p><p><em>Kenneth Styppa is part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at BMW and currently pursues his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jonas Wahl is a postdoctoral researcher within the research group Climate Informatics at TU Berlin. He obtained his PhD in mathematics at KU Leuven (Belgium) and has worked at the Hausdorff Centre for Mathematics in Bonn before joining Jakob’s group in TU Berlin. His research focuses on causal inference for high-dimensional spatiotemporal data. You can read more about Jonas on his personal website </em><a href="https://jonaswahl.com/"><em>https://jonaswahl.com</em></a><em>.</em></p><p><em>Jakob Runge heads the Causal Inference group at </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em> in Jena and is chair of computer science at TU Berlin. The Causal Inference group develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.climateinformaticslab.com/"><em>www.climateinformaticslab.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ab49259e921" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/structural-causal-models-a-quick-introduction-1ab49259e921">Structural Causal Models — A Quick Introduction</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Conditional Independence and Causal Discovery]]></title>
            <link>https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/77919db6159c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 09:16:07 GMT</pubDate>
            <atom:updated>2023-06-13T09:16:07.611Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MykUG2OA-PkS9I6CJyFWoA.png" /></figure><h4>A Gentle Guide to Causal Inference with Machine Learning Pt. 6</h4><p><em>“Why?” has been a central question of humanity and the birthplace of science hundreds of years ago. The human will to make sense of their surroundings was its fuel, experiments and theories formulated in mathematics were its motor since Galileo Galilei developed the scientific method at the end of the 16th century. But what happens if experiments are simply impossible? What happens if all we have are observational data? Are we doomed to ground our models and worldview in correlations instead of underlying mechanisms at play?</em></p><p><em>Luckily, we are not! The goal of Causal Inference is to do exactly that: learning causal relationships just from observational data.</em></p><p><em>You should be familiar with the following concepts from the previous articles before you keep on reading:</em></p><ol><li><a href="https://medium.com/causality-in-data-science/assumptions-for-causal-discovery-cc194d607a14"><em>Causal Markov, Faithfulness and Causal Sufficiency Assumption</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-is-causal-machine-learning-ceb480fd2902"><em>Fundamental idea behind causal inference</em></a></li><li><a href="https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a"><em>Causal Graphs</em></a><em> &amp; </em><a href="https://medium.com/causality-in-data-science/confounding-colliding-d-separation-and-sleeping-with-shoes-on-8ba43c976354"><em>D-separation</em></a></li></ol><p>To answer the causal questions that humans ask all the time, scientists have come up with two ways.</p><ol><li>Performing experiments or experimental studies: Conducting so-called interventions to assess causal relationships. E.g. giving Group A a medication, Group B a placebo, and Group C no medication to assess the effect of medication on some health outcome. Or designing an experiment in physics.</li><li>Performing observational causal inference studies: Using assumptions in a theoretical framework to derive causal statements from a data sample of an observational distribution</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0z5z_ygw5-fxVoSmn0oNEg.png" /></figure><p>Although experimental studies such as Randomized Control Trials are the gold standard for identifying causal relationships in a number of fields, such as in medicine, running these is often not feasible. Consider the example of climate sciences. We only have one planet. Performing climate change experiments on this planet is either impossible or highly unethical. Confronted with these limitations, causal inference gives us a method to identify causal relationships nevertheless. Sounds like magic? It’s not, it’s just math, logic, and reasoning about assumptions of the underlying system. Let’s understand how this works.</p><h4>The two tasks of causal inference</h4><p>In essence, there are two tasks in causal inference and their difference resides in their distinct starting point.</p><p>Task 1 uses already existing qualitative knowledge about causal relationships to build a directed acyclic graph which is then used to quantitatively estimate causal effects. Here we refer to this task as causal effect estimation.</p><p>Task 2, on the other hand, starts from a blank slate. You have no clue about how your variables could be related and therefore go on an adventurous discovery journey to find out. This task is also called causal discovery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fVYYMY9WP6Va9eMX" /></figure><h4>Causal Discovery and Conditional Independence</h4><p>By Reichenbach’s common cause principle, we can assume that if X and Y are statistically related (=correlated) and the data is unbiased, then</p><ol><li>X is either the cause of Y, or</li><li>Y is the cause of X, or</li><li>there is a common cause Z that causes both X and Y.</li></ol><p>Sounds reasonable, right? More generally, this is called the Causal Markov Condition which states that statistical dependencies emerge from causal relations. The converse of the Markov assumption is the so-called Faithfulness assumption which states that statistical independencies in the distribution imply graphical d-separation in the graph.</p><p>We will now see that if both of these conditions are fulfilled, we get a highly useful direct relation between causal connections in the graph and statistical dependencies in the data that we can use in order to discover a representation of the underlying causal mechanism.</p><p><strong>Now let’s assume we have observational data from three variables and no idea about their true causal relationships</strong>. Fortunately, we can do something about that. We use statistical tests for conditional independence to test whether there is any kind of relationship between X and Z. If no dependence is found, that is, the two are statistically independent, then the Faithfulness condition tells us that there is no causal relationship and we can remove the edge from our graph. If they are not independent, our journey of causal discovery has just begun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T0iYpGm9LFm17yWW" /></figure><h4>Independence-based Causal Discovery with the PC algorithm</h4><p>As a result of the rigorous development of a mathematical framework around causal inference, several causal discovery algorithms have been developed. Being one of the first algorithms published for causal discovery, the PC algorithm as introduced by Peter Spirtes and Clark Glymour (named after their first names) is not only one of the most popular and still widely used algorithms, but also provides an excellently intuitive entry point into causal discovery. Thus, we choose the PC algorithm as an example to illustrate the process of causal graph identification. The PC algorithm makes one further assumption next to the Causal Markov Condition and Faithfulness: It assumes that there are no unobserved confounders, called the Causal Sufficiency assumption. However, there exist other algorithms that allow to deal with hidden confounding, but let’s keep it simple for now.</p><p>This is how the PC algorithm works:</p><ol><li>Make a node for each observed variable</li><li>Start with all of them being connected to each other.</li><li>Eliminate as many edges as possible using conditional independence tests. More specifically, remove edges X-Y if X is independent of Y given a conditioning set S. Step 3 is a repetitive procedure, starting with S as the empty set S={} and increasing its size (cardinality) by 1 for every iteration.</li><li>Establish (causal) directions for each remaining edge using colliders, the assumption that there are no cycles, and any other assumptions you can make use of, such as time order.</li></ol><p>Let’s take a look at each step in an example.</p><p><strong>Skeleton Identification</strong></p><ol><li>We are given a data set with five variables ABZED (see figure below). As described before, we start with the fully connected graph. As a first step towards identifying the potential causal relations, we test for independence conditioning on the empty set. As can be seen in the true causal graph (marked as green) using the <a href="https://medium.com/causality-in-data-science/confounding-colliding-d-separation-and-sleeping-with-shoes-on-8ba43c976354">rules of d-separation</a>, A and B are independent. Consequently, the edge between A and B is removed.</li><li>Next, we test for conditional independence using a conditioning set containing one node, for example, S={Z}. This leads to removing the edges A-E, A-D, E-D, B-D, and B-E. Performing more conditional independence tests conditioning on A or B will not lead to removing any more edges. Thus, we have converged to what we assume to be the actual skeleton of the true causal graph. The PC algorithm chooses the conditioning sets S in a smart way by only selecting subsets of adjacent nodes of the two nodes to be tested.</li><li>Converged towards this graphical representation, we uncover the causal directions. One way to achieve this is quite obvious. It is the exploitation of time dependencies. If A happens before Z, the arrow goes from A to Z. If this is not possible, collider-structures can help.</li></ol><p><strong>Collider Orientation and further orientation rules</strong></p><p>The algorithm now proceeds by considering all unshielded triples in the skeleton. A triple such as A-Z-B is unshielded if</p><p>(i) the edge A-B has been removed in the previous PC skeleton phase, and</p><p>(ii) there are links between A-Z and Z-B.</p><p>Now the collider rule says that if Z was not in the conditioning set (also called separating set) of the statistical test by which the link between A and B was removed in the skeleton discovery phase, then the arrows can only point to Z.</p><p>Why? Because otherwise there would be a dependence between A and B (by d-separation), but we didn’t measure any. Indeed this is the case as we can see in the true green causal graph.</p><p>Now this oriented collider structure helps us in determining the other orientations. For example, no such collider can be found for A-Z-E or any other remaining triple. What does this tell us? If there would be a directed edge from E to Z, then A and E should be independent given the empty set S={}. But in the skeleton phase we found them to be independent given Z. So, we can assume that the arrow goes from Z to E because this well explains the observed (in)dependenvies. The same accounts for the edge from Z to D. This way, all other remaining edges can be directed and we discovered the true causal graph on the right. Mission fulfilled!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lv9oNwpKkxbk0U0F" /><figcaption>The example is from: <a href="https://www.bradyneal.com/Introduction_to_Causal_Inference-Dec17_2020-Neal.pdf">https://www.bradyneal.com/Introduction_to_Causal_Inference-Dec17_2020-Neal.pdf</a></figcaption></figure><h4>When does this work?</h4><p>It is important to note that only certain mathematical assumptions render these causal dependencies identifiable. Here we used the Causal Markov, Faithfulness and Causal Sufficiency assumption to identify the causal structure. If they are not met, some of the inferential power will be lost. But with new problems, new solutions and methods are developed such as a version of the PC-algorithm that does not assume Sufficiency called the FCI algorithm.</p><p>Consequently, causal inference is an ongoing process, testing the eligibility of certain assumptions and selecting/developing the appropriate method for the given task.</p><p>For more details on the most fundamental assumptions of causal discovery have a look at the previous blogposts as linked at the beginning.</p><p>So far so good for an introduction. More recent algorithms follow in the next articles. Thanks for reading!</p><p>About the authors:</p><p><em>Kenneth Styppa is part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at BMW and currently pursues his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jakob Runge heads the Causal Inference group at </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em> in Jena and is chair of computer science at TU Berlin. The Causal Inference group develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.climateinformaticslab.com/"><em>www.climateinformaticslab.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77919db6159c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/introducing-conditional-independence-and-causal-discovery-77919db6159c">Introducing Conditional Independence and Causal Discovery</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are Causal Graphs?]]></title>
            <link>https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/abdb50354c8a</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[graphical-model]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Tue, 14 Mar 2023 10:33:57 GMT</pubDate>
            <atom:updated>2023-03-14T10:33:57.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x30h9LNicZ-8ioS_7yEzEg.png" /></figure><h4>A Gentle Guide to Causal Inference with Machine Learning Pt. 3</h4><p><em>Sometimes we need to give things a visual structure to understand them. That’s not only true for learning how brain cells work in school. It is also true for understanding any kind of phenomena. In causal inference, the most important way to depict causal relationships is via so-called Causal Graphs. So let’s use the next article and family Skywalker to understand them. May the force be with you!</em></p><h4>Basic Graphs</h4><p>In simple words, a graph is a network. You have nodes <strong><em>V, </em></strong><em>s</em>ay <strong><em>V </em></strong>= <em>{X, Y, Z</em>}, and you have edges <strong><em>E</em></strong>. The edges connect two nodes, while each node represents a (random) variable. Edges and nodes together make up a graph <em>G = (</em><strong><em>V, E</em></strong><em>) </em>that can be depicted as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJ39RsSNWPu-kuXfL-zrIA.png" /></figure><p>There are many examples of graphs. For example, the instagram network could be depicted by a graph, with the users being the nodes and the edges representing whether these people are friends or not. Such big graphs tend to look more beautiful than the simple ones. But beware, the more beautiful they are, the more complex it gets to understand their relations!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*nIc_PDZJcsdep7OF" /><figcaption>(Baird et al., 2012)</figcaption></figure><h4>From Graphs to Node Relationships: Directed Acyclic Graphs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/0*pJAtsiaZI2hrE_4X" /></figure><p>To represent causal relations, we need to move from undirected to directed graphs. Then we can express relations between nodes by kinship relations and speak of parents, children, ancestors, and descendants.</p><p>Let’s use <em>i </em>and <em>j </em>to indicate two nodes in a graph. Then node <em>i</em> is a parent of <em>j</em> if the graph contains the edge <em>E(i,j)</em> but not <em>E(j,i)</em>. In other words, node <em>i</em> is a <em>parent</em> of <em>j </em>when there is an edge from <em>i</em> to <em>j</em> but not the other way around (Darth Vader for example “caused” Luke Skywalker to be there but not the other way around). Node <em>j </em>is then called a child.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjIqWiXNh81EWlF3vM6qiQ.png" /></figure><p>Node <em>i </em>and<em> j</em> are also called to be <em>adjacent</em> which means that they are connected. If all nodes of <em>G</em> would be adjacent, then we would have a fully connected graph.</p><p>As you probably noticed, the concept of direction is necessary for defining parents and children. If all edges in <em>G </em>have such directions we speak of a directed graph. If you encounter the term “skeleton”, this is the simplified graph that does not take the directions into account. Thus, there is no distinction between two graphs with different directions if the same nodes are adjacent.</p><p>A more specific version of a directed graph is a directed acyclic graph, also called DAG. It can be characterized as a directed graph without directed cycles. Simply speaking, there is no loop in the system (e.g., altitude causes temperature but not also the other way around). The rationale behind this restriction is that trying to identify a reason prohibits the reason from being the reason of itself, otherwise we could not identify a reason (makes sense, right?). We will learn more about cyclic graphs and how they often can be resolved in later blog posts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w_SyaldgXzSQCG5T" /></figure><p>Nodes do not have to be directly connected through one edge. Sometimes they are connected through several edges (similar to Luke Skywalker being connected to Shmi Skiwalker, his grandmother). The list of distinct nodes you “walk” along from <em>i</em> to <em>j</em> is called a path. This path can include several edges or just one, but a path has no repeated nodes. If the edges on a path are all oriented from <em>i</em> to<em> j</em>, <em>i</em> is an ancestor of <em>j</em> and <em>j</em> a descendant of <em>i</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0f0O5cisJ_YYtu5b" /></figure><h4>From DAGs to Causal Graphs</h4><p>What makes a DAG a causal graph? Well, essentially it is the interpretation that we give to the directed edges. If we assume that they represent a causal relationship between the connected nodes along with the direction of the edge, we found a visual representation of a causal relation (e.g. Shim Skywalker directly causing Darth Vader and indirectly Luke Skywalker). This makes causal graphs very powerful, as they provide us with an intuitive way to represent complex causal phenomena, while also helping us make the distinction between causal and non-causal associations.</p><p>It is important to note that every edge in a causal graph indicates a direct and no indirect causal effect or correlation through other nodes of the graph. This not only makes our graphs less complex, but also enables us to model the effect of interventions (read more on interventions in another article).</p><p>For now, let’s call it a blog post. In the next one, we will discuss important concepts of relationships within causal graphs and how they influence the process of uncovering causal relationships. More specifically we will discuss Simpson’s paradox, confounding, colliders, mediators, and d-separation, before moving on to some of the most important assumptions that help us identify causal effects in an observational setting. Follow each blog post step by step. It will get more and more intriguing.</p><p>Thanks for reading!</p><p>About the authors:</p><p><em>Kenneth Styppa is part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at BMW and currently pursues his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jakob Runge heads the Causal Inference group at </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em> in Jena and is chair of computer science at TU Berlin. The Causal Inference group develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.climateinformaticslab.com/"><em>www.climateinformaticslab.com</em></a></p><p>Picture References:</p><p>Oleson, E. M., Boggs, C. H., Forney, K. A., Hanson, M. B., Kobayashi, D. R., Taylor, B. L., … &amp; Ylitalo, G. M. (2012). Reevaluation of the DPS designation for Hawaiian (now Main Hawaiian Islands) insular false killer whales. <em>Honolulu, HI</em>.</p><p><a href="https://www.google.com/search?q=I+am+your+father+meme&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjQy4exrIv9AhUO-qQKHTf2BNAQ_AUoAXoECAEQAw&amp;biw=1293&amp;bih=739&amp;dpr=2#imgrc=VUcp8baRiWBCRM">I am your father meme - Google Search</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abdb50354c8a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/what-are-causal-graphs-abdb50354c8a">What are Causal Graphs?</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Machine Learning Needs Causality]]></title>
            <link>https://medium.com/causality-in-data-science/why-machine-learning-needs-causality-3d33e512cd37?source=rss-42868c07c23b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d33e512cd37</guid>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jakob Runge]]></dc:creator>
            <pubDate>Mon, 13 Feb 2023 20:10:03 GMT</pubDate>
            <atom:updated>2023-02-13T20:10:03.439Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WbBn4w91VB0brTXlSBNSWw.png" /></figure><h4>A Gentle Guide to Causal Inference with Machine Learning Pt. 1</h4><p><em>When computers fail and natural intelligence excels, we stand at the boundary of current machine learning research. But what makes animals so superior? Why is it possible that we can identify a tree as a tree no matter the light or image quality and why is it so challenging for ML algorithms to adapt to these changes? The answer lies in the difference between correlation and causation. In this blog series, we give a gentle introduction for newcomers to causal inference.</em></p><p>Inferring knowledge about causal relationships is at the heart of the scientific enterprise and the field of causal inference is about learning those from observational data. In today’s data-driven world causal inference hopes to lift machine learning onto its next level, solving many of the hardest and seemingly ever-existing problems such as overfitting and prediction under new conditions.</p><p>In the end, machine learning boils down to a large pattern recognition system, which leverages the power of associations and correlations in the setting of independent and identically distributed data (i.i.d). In other words, machine learning sees things, assumes them to be true relationships, and extrapolates its decisions on the future based on these past experiences.<br>This recognition and exploitation of associations leads to undoubtedly powerful predictions, yet it encounters a fundamental challenge. Real-world data often does not fulfill the conditions necessary for stable prediction outcomes across scenarios. In computer vision for example, the test set distribution might differ fundamentally from the training distribution due to changes in the light conditions, the quality of the camera, or its viewpoint. These changes, however, contain the potential for completely defeating or at least heavily impacting the accuracy and usefulness of the trained model.</p><p>On the other hand, understanding the mechanism between a cause and its effect enables causal models to cover a range of distributions, one for each possible intervention, or perform reliable predictions under a change in the environment. In contrast, purely statistical learning models, which are based on the i.i.d.- assumption, only allow for one general population distribution and do not offer this flexibility, making them less generalizable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RvRhbZeq479CyOw3" /></figure><p>As predictions find increasing implementations in human decision-making, this well-known generalization problem of association-based machine learning goes beyond worse accuracy levels. It might result in fundamentally wrong decisions or conclusions about reality. A machine-learned system that detects the positive association between the European stork population and birth rates might result in a powerful prediction algorithm for birth rates. However, changing the stork population in a country will obviously not affect birth rates even though purely statistical models may predict so. <br>Of course, data scientists would never train such a model you might think. And this is true for the stork example. But indeed there are many situations where our judgment or intuition fails. The crucial point is that we don’t know when this is the case.</p><p>Let’s have a look at this with an example. Consider the following equations with three variables and jointly independent noise terms as a given ground truth system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZ8vA8msNxiBXpz3ZTw_ow.png" /></figure><p>In reality, we do not know these systems, we call them structural causal models (SCMs). All we have (besides some theory) is observational data from the system. So let’s act as if this was the case and generate a sample from the joint distribution <em>(X, Y, Z)</em> as our “observations”.</p><pre># Defining the systems variables<br>import numpy as np<br>import random<br>import matplotlib.pyplot as plt<br>random.seed(10)<br>Z = np.random.normal(loc = 3, scale=1, size = 300).reshape(-1, 1)<br>Y = 3*Z + np.random.normal(loc = 1, scale=0.5, size = 300).reshape(-1, 1)<br>X = 2*(Z-1) + np.random.normal(loc = 3, scale=1, size = 300).reshape(-1, 1)</pre><p>Given this information, we want to get a solid understanding of the underlying system. Say, by plotting our data we see how our variables <em>X</em> and <em>Y</em> relate to each other.</p><pre># Visualize the association<br>plt.scatter(X, Y, color=&#39;blue&#39;)<br>plt.xlabel(&#39;X&#39;)<br>plt.ylabel(&#39;Y&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*oip-z0LQNgMWUXxVjKiEKQ.png" /></figure><p>Consequently, we could fancy that those two share a connection that we want to explore with a basic regression. Regressing <em>Y</em> on <em>X</em> will consequently result in a decent prediction model although no actual direct functional relation between <em>X</em> and <em>Y</em> exists (you can check that this is the case in the ground truth SCM system written above). Note, that this is independent of the kind of ML algorithm we use. If we only have the information above (<em>X</em> and <em>Y</em>), even the most powerful models will learn and use a relationship for their predictions that is not a causal one.</p><pre># Regress Y on X<br>from sklearn.linear_model import LinearRegression<br>linreg = LinearRegression()<br>result = linreg.fit(X, Y)<br><br># Evaluate the regression<br>r_sq = result.score(X, Y)<br>print(r_sq) #output:0.78214..</pre><p>If we would be able to intervene in the system and use the resulting interventional data, we could uncover such false conclusions.</p><p>Assume we can do that. Below, we modify our system by intervening on <em>X</em>.</p><pre># The effect of intervening on X<br>np.random.seed(10)<br>Z = np.random.normal(loc = 3, scale=1, size = 300).reshape(-1, 1)<br>Y = 3*Z + np.random.normal(loc = 1, scale=0.5, size = 300).reshape(-1, 1)<br>X = np.random.normal(loc = 3, scale=1, size = 300).reshape(-1, 1) #intervene on X</pre><p>If <em>X</em> and <em>Y</em> would have a true causal connection, the intervention on <em>X</em> should have an impact on <em>Y</em>. The mechanism that connects <em>X</em> and<em> Y</em> should be invariant. However, when performing a second regression on this interventional data we can make a totally different observation:</p><pre>linreg2 = LinearRegression()<br>result2 = linreg2.fit(X, Y)<br># Evaluate the regression<br>r_sq_2 = result2.score(X, Y)<br>print(r_sq_2) #output: 0.0025...</pre><p>As can be seen above, we have an extremely low R-squared value, indicating a very poor fit of our regression. In other words, there seems to be no connection between <em>X</em> and <em>Y</em>.</p><p>This gets even more obvious when plotting the data:</p><pre># Visualize the association<br>plt.scatter(X, Y, color=&#39;blue&#39;)<br>plt.xlabel(&#39;X&#39;)<br>plt.ylabel(&#39;Y&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*p7swg9PUxog0daUvkKMzPA.png" /></figure><p>Imagine, you trained a model that performs its predictions based on the false relationship detected earlier and is now confronted with the above change in its environment. It is doomed to produce rubbish outcomes.</p><p>A quick answer to this problem could be: “Then let&#39;s do as many experiments, and/or feed our model with as much data as possible, hoping the model will account for all eventualities and gain a reliable understanding of its application context”.</p><p>And in some cases, it can work. However, such a strategy is limited to fields with extremely large data capacities and no major risk of distribution shifts (e.g. NLP). In all others, it is either monetarily infeasible or simply morally unacceptable to conduct any kind of experiment. This gives us a large space of cases in which machine learning stands on a weak foundation and no experiments are possible.</p><p>That’s exactly where the toolkit of causal inference comes into play. Although being unable to intervene directly in the system, <strong>we can use expert assumptions and algorithms to achieve a similar effect that in the end allows us to obtain a deep understanding of the causal system we observe</strong>.</p><p>So what’s the moral of the story? Simply using our data as it is, performing some feature engineering, and feeding them into powerful algorithms might lead to good results, but these results could stand on shaky ground since no causal knowledge of the actual system is obtained. Incorporating the tools of causal inference into machine learning brings us further.</p><h4>Causal Inference Powered Machine Learning</h4><blockquote>„If we wish to incorporate learning algorithms into human decision making, we need to trust that the predictions of the algorithm will remain valid if the experimental conditions are changed.“ (Schölkopf et al., 2021).</blockquote><p>From this perspective, the current evolution of machine learning towards grounding models in fundamental underlying structures rather than correlation-based associative links is not altogether very surprising. This fundamental underlying structure can be offered by causal inference, a set of mathematical tools to answer questions about causal relationships based on observational or experimental data by utilizing assumptions about the underlying system.</p><p>Causal inference can be distinguished into two fundamental tasks.</p><ol><li>Utilizing qualitative causal knowledge for making causal predictions</li><li>Learning causal graphs (causal discovery)</li></ol><p>Causal machine learning emerges at the crossroads of conventional machine learning and causal inference. On the one hand, identifying the causal graph of a prediction problem provides essential support for better machine learning models. On the other hand, established machine learning methods can be utilized within the framework of causal inference to learn causal relations or even causal variables.</p><p>Both can be understood as <em>Causal Machine Learning</em>.</p><p>In this series of blog posts, we will give a gentle introduction to this growing and quickly developing field of new research — starting with the basics first. Join us on our journey and discover how models might look to provide answers to the seemingly simple question of “Why”.</p><p>About the authors:</p><p><em>Kenneth Styppa is part of the Causal Inference group at the </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em>. He has a background in Information Systems and Entrepreneurship from UC Berkeley and Zeppelin University, where he has engaged in both startup and research projects related to Machine Learning. Besides working together with Jakob, Kenneth worked as a data scientist at BMW and currently pursues his graduate degree in Applied Mathematics and Computer Science at Heidelberg University. More on: </em><a href="https://www.linkedin.com/in/kenneth-styppa-546779159/"><em>https://www.linkedin.com/in/kenneth-styppa-546779159/</em></a></p><p><em>Jakob Runge heads the Causal Inference group at </em><a href="https://www.dlr.de/dw/desktopdefault.aspx/tabid-12192/21400_read-49437/"><em>German Aerospace Center’s Institute of Data Science</em></a><em> in Jena and is chair of computer science at TU Berlin. The Causal Inference group develops causal inference theory, methods, and accessible tools for applications in Earth system sciences and many other domains. Jakob holds a physics PhD from Humboldt University Berlin and started his journey in causal inference at the Potsdam Institute for Climate Impact Research. The group’s methods are distributed open-source on </em><a href="https://github.com/jakobrunge/tigramite.git."><em>https://github.com/jakobrunge/tigramite.git.</em></a><em> More about the group on </em><a href="http://www.climateinformaticslab.com/"><em>www.climateinformaticslab.com</em></a></p><p>Quoted from:</p><p>Schölkopf, Bernhard, et al. “Toward causal representation learning.” <em>Proceedings of the IEEE</em> 109.5 (2021): 612–634.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d33e512cd37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/causality-in-data-science/why-machine-learning-needs-causality-3d33e512cd37">Why Machine Learning Needs Causality</a> was originally published in <a href="https://medium.com/causality-in-data-science">Causality in Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>