BICauseTree: Bias-balancing Interpretable Causal Tree

A model for interpretable estimation of population-level causal effect that can handle high-dimensional datasets.

Causal Inference

What is Causal Inference ?

Inference: Learn about what you do not observe (parameters) from what you do observe
Causal Inference: Inferring the causal effect of any treatment/policy/intervention, especially from observational data. Examples include: impact of a public policy, effect of a treatment on a disease
Here, we only consider binary treatment/interventions

Why is Causal Inference important ?

• Investigating causal relationships: natural human reasoning
• Often, the treatment/intervention can’t be randomized e.g.\ smoking or a public health policy
• Observational data has confounding bias, see example below

Is chemotherapy a protective factor against death from cancer?

Figure 1. DAG of a confounding bias

Intuitively, we would consider the Average Treatment Effect (ATE) to be equal to:
𝔼 (Y|T=1) — 𝔼 (Y|T=0)
However, this would give us a biased estimate of treatment effect as receiving chemotherapy is associated with both having a more severe cancer, and higher odds of death.

ATE = 𝔼 (Y(1)) — 𝔼 (Y(0))
where Y(t) is the outcome vector had all individuals received treatment t, commonly referred to as the potential outcome [1].

Why is Causal Inference challenging ?

• To mitigate confounding bias, we need to compute the counterfactual outcome i.e. the potential outcome in the opposite treatment scenario to the observed
• As both potential outcomes cannot be observed, we lack ground truth and therefore, both effect estimation and model evaluation are difficult

Why should causal estimators be interpretable ?

Let’s first explicit the difference between an interpretable model and an explainable model.

Interpretable: model decisions are explicit and traceable
Explainable: model decisions are justified post-hoc using an external explanation model

However, explanation models have severe limitations. Furthermore, a growing part of the scientific community [2] has expressed skepticism about using complex black-box models that need an explanation model on top of them, instead of using simpler interpretable predictive models altogether. Finally, Causal Inference requires all the more transparency as it applies to safety-sensitive fields, e.g. healthcare, policy-making, resource allocation. We therefore advocate for interpretable causal estimators.

What is the state of the art lacking ?

Existing causal estimators are either:

• complex models which perform well but aren’t interpretable e.g.\ adjustment methods
• interpretable models which may be biased and cannot handle high-dimensional date e.g.\ matching, partitioning

→ We introduce BICauseTree: Bias-balancing Interpretable Causal Tree.
A model for unbiased, interpretable estimation of Average Treatment Effects (ATEs) that can handle high-dimensional datasets.

BICause Tree

Intuition and overview

BICause Tree is a binary decision tree that stratifies on imbalanced covariates w.r.t treatment allocation.

By recursively splitting on the feature with the highest Absolute Standardized Mean Difference (ASMD) between the treated and untreated, we may find clusters where treatment allocation is less biased than in the initial overall cohort.

Therefore, BICause Tree is able to identify the subpopulations with similar propensities to be treated i.e. natural experiments. It can further identify positivity violations i.e., subsets of the population where treatment allocation is highly unbalanced directly in the covariate space.

Algorithmic structure

(1) Building the tree

• Find the feature j with maximum ASMD
• We then split on the feature value x_split_j with the highest association to treatment assignment. We do a Fisher test/chi² test and take the feature value with the lowest p-value.
• Store the p-value of the resulting split (is it a significant difference?)
• Split the data and repeat until the stopping criterion is met

Examples of stopping criteria include: maximum ASMD, maximum depth, node size, treatment group size…

(2) Pruning the tree

• Do multiple hypothesis test correction on the p-values of all splits
• Prune to keep significant splits or/and splits with at least one significant split amongst their descendants

Here, the idea is to avoid stopping too early if p-values get more significant further down.

Get Lucile Ter-Minassian’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

(3) Positivity violation filtering

• Compute upper/lower thresholds for propensity weights deemed extreme
• Within each leaf: check if prevalence is above/beyond threshold

Leaf nodes with positivity violations are excluded. These subpopulations are not considered when computing the Average Treatment Effect.

As a result, the overall algorithmic structure of BICauseTree is as follows:

1. Building the tree. While stopping criteria unmet:
   — Compute ASMD for all possible splits
   — Split according to maximum ASMD
   — Record p-value of all \textit{splits}
2. Do multiple hypothesis test correction of split p-values. Prune tree.
3. Filter leaf nodes that violate positivity

Figure 2. BICause Tree partition on a simplistic simulated dataset with three binary variables. N: node sample size, P: node prevalence. Leaf nodes outlined in red violate positivity.

ATE estimation purposes

Once the tree is built (i.e. the partition) we can either estimate the counterfactual outcomes as the average outcomes of the treated/untreated samples in each leaf or fit an arbitrary causal model (e.g., IPW, outcome regression) within each leaf. Similarly, one can use the leaf prevalence as a propensity, or fit a more complex propensity score model.

Causal effects are estimated by taking a weighted average of the local treatment effects in each subpopulation (or leaf node).

Overall, we would typically expect users to partition their data into a training and testing set.

At the training phase, the data would be used to first fit the partition, and then fit the leaf-specific outcome/propensity models if relevant.

At the testing phase, the data would be sent down the tree, and then used to estimate potential outcomes using the previously fitted leaf-specific outcome models.

Other estimation purposes

Alternative use-cases for BICauseTree include using the partition as a propensity model, or as an outcome model. Again, the data can be sent down the tree and used for estimation at the testing phase.

Implementation

model = BICauseTree(min_treat_group_size=2, asmd_threshold_split=0.1, 
  max_depth=10, individual=False, multiple_hypothesis_test_alpha=0.05,
  multiple_hypothesis_test_method='holm', positivity_filtering_method='crump',
  positivity_filtering_kwargs=positivity_filtering_kwargs,
  outcome_model=MarginalOutcomeEstimator(learner=None))

model.fit(X, t, y)

effects = model.estimate_population_outcome(X, t, y)

Results on a realistic causal benchmark dataset

The twins dataset

The twins dataset is based on real-world records of 11,984 pairs of same-sex twin births, and has 75 covariates. It is based on real-world records of $N = 11,984$ pairs of same-sex twin births, and has 75 covariates. It tests the effect of being born the heavier twin (i.e.\ the treatment) on death within one year (i.e.\ the outcome), with the outcomes of the twins serving as the two potential outcomes. In other words, the twins dataset allows for estimation of the true causal effect. We use the dataset generated by Neal et. al [3], that simulates an observational study from the initial data by selectively hiding one of the twins with a generative approach.

Identifying natural experiments and estimating causal effect

Figure 3. Estimation bias for the twins dataset (N = 11,984) across 50 subsamples, excluding positivity violating leaf nodes.

Figure 3 shows the distribution of the estimation biases across subsamples on the twins dataset, comparing to the baseline models. Here, we use both the simplest version of our tree which we term BICauseTree(Marginal) where the effect is estimated from taking average outcomes in leaf nodes. and BICauseTree(IPW), an augmented version in which an IPW model is fitted in each leaf node. To compare estimation methods, we compute the difference between the estimated ATE and the true ATE for each subsample (or train-test partition) and display the resulting distribution of estimation biases in a box plot.

In this experiment, BICauseTree(Marginal) estimator is less biased than the marginal estimator. Augmenting our tree with an IPW outcome model — BICauseTree(IPW) — further decreases estimation bias, making it comparable with IPW, both w.r.t bias and estimation variance.

Note that prior to this, we had also shown BICause Tree’s ability to identify natural experiments on simulated data where the treatment allocation mechanism was known. See our paper for further details.

Filtering positivity violations directly in the covariate space

Figure 4. BICauseTree partition for the twins dataset

Figure 4 above shows the BICauseTree partition obtained. The leaf node outlined in red is considered to be violating positivity. Therefore, in this experiment, we claim that our estimate of the effect of being born heavier might not be valid for newborns that fit the criteria for this specific violating node.

This capability of BICauseTree is highly valuable in any safety-sensitive setting. Consider a scenario where the ‘at-risk’ twin benefits from a follow-up visit after birth, and that the true effect of the intervention is higher in the positivity violating subpopulation. Extrapolating the estimated effect of the treatment to the entire cohort may be dangerous to the infants in this subgroup.

It is thus essential for practitioners to know which population the inferred effect applies to. This would not have been possible using alternative non-interpretable methods for identifying positivity violations e.g. IPW with weight trimming, as they provide an opaque exclusion criterion. Additionally, note that the positivity violation identification remains transparent regardless of the chosen propensity or outcome model at the leaves.

Bias-interpretability tradeoff

Figure 3 shows how augmenting our partition with an IPW leaf outcome models has decreased the estimation bias. The gap between BICause(Marginal) and BICause(IPW) thus represents the need for a more complex outcome model in the leaves, or the estimation bias that was traded against interpretability here.

Strengths and limitations of our approach

Strengths

• Ability to find natural experiments when they exist
• Interpretable estimation while maintaining performance comparable to existing methods such as IPW
• Safety and applicability of having a transparent positivity filtering mechanism
• Non-parametric estimator: flexible adaptation to different functional forms

Limitations

• Lower data efficiency than most other estimators: trees are very data greedy
• Splits do not consider interactions, only the marginal association of covariates with T
• Binary treatment only

This work was done under a dual Oxford University - IBM Research affiliation. IBM Research has filed this novelty to the United States Patent and Trademark Office.

[1] Rubin DB. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association. 2005 Mar 1;100(469):322–31.

[2] Rudin C. Stop explaining black-box machine learning models for high-stakes decisions and use interpretable models instead. Nature machine intelligence. 2019 May;1(5):206–15.

[3] Neal B, Huang CW, Raghupathi S. Realcause: Realistic causal inference benchmarking. arXiv preprint arXiv:2011.15007. 2020 Nov 30.