Why Your Marketing Model Is Broken

Sample, label, target, evaluation, attribution — the five links where marketing models quietly break, and why randomization is the investment that comes before any model.

A retention campaign that raised churn

In 2007, Nicholas Radcliffe and Patrick Simpson of the UK consultancy Stochastic Solutions published a white paper on customer-retention campaigns at several telecom operators. At one of them, anonymized as Operator 2, the campaign’s actual result was this: the control group, which received no retention offer, churned at 9%; the treatment group, which got the offer, churned at 10%. They spent the budget, made the calls, and more customers left.

The authors didn’t stop at that headline number. They segmented customers by churn risk and checked the treatment-minus-control gap segment by segment, and derived a second result: even if you narrowed the campaign to only the highest-risk 20% picked out by the churn model, overall churn would still land about 1 percentage point above doing nothing at all. The first number is what actually happened; the second is a targeting scenario derived from the same experimental data. Both target the customers the churn model flags as highest-risk. So where does it go wrong?

Every step on this pipeline checks out in isolation. The churn model is trained properly and ranks fine; targeting follows the score strictly; the offer really does reach the customer. The authors’ explanation is that the highest-churn-risk customers are not necessarily the ones retention works on. Some of them had already half-forgotten the operator and wouldn’t have bothered to switch at contract’s end, and a retention call reminds them to go check what the competition charges. The line the paper left as a warning has been cited by the uplift-modeling community for close to twenty years: “Targeting on the basis of predicted churn risk is a dangerous strategy.”

Retention is just one kind of campaign. Coupons, push notifications, SMS, outbound calls, membership perks. The modeling behind all of them looks the same in outline: pull historical data, train a response model, rank people by score, and spend the budget on the highest-scoring ones. Operator 2 shows that every step of this pipeline can be right and the whole still get the job exactly backwards. Look closely, and the problem runs the length of the chain.

There are five links here, from the first modeling step to the last, and each one has its own way of going wrong:

• Sample. The population you train on isn’t the population you score. The model learns on users who were targeted in the past but has to score everyone, so the data is biased from day one.
• Label. Non-converters get filed as negatives wholesale, but that pile mixes in people who were never reached, never saw the message, or converted after the attribution window closed. A non-converter is not necessarily an uninterested customer.
• Target. What the campaign actually wants to optimize is incremental lift, and no individual’s lift is ever observable — the same person can’t both receive and not receive the coupon.
• Evaluation. Uplift has no ground truth; offline metrics only tell the truth on randomized traffic.
• Attribution. A dozen campaigns run at once, and the same conversion gets booked into several reports at the same time.

For the past twenty-odd years, the through-line of algorithmic progress in marketing modeling has been patching these five links one at a time. And every patch lands on the same thing in the end: randomized traffic, and the measurement built on top of it. That priority is the exact reverse of most teams’ default order.

Sample: biased, thin, and drifting

A response model’s training data comes from the logs of past campaigns. And past targeting was never random: last cycle’s list was drawn by the previous model or by an operations rule. So the training set is all “people who were targeted before.” The model learns the regularities of that group and then has to score the full population. The machine-learning literature calls this sample selection bias; Zadrozny formalized it in 2004: whether a sample makes it into training is governed by a selection variable that itself depends on user features, and some learners will simply learn the wrong thing on data like this. What makes marketing special is that the selection variable is the previous model itself.

The structure also reinforces itself. Each cycle targets by last cycle’s score, new labels land only on the people who were targeted, so the next model has only ever seen that group, and the cycle after targets by its score. Research on decision-system logs calls this logged bandit feedback: the log records only “the response after we acted on the people we chose,” and the people we didn’t choose leave no row at all. Swaminathan and Joachims showed that naive supervised learning on logs like these is systematically biased. The model’s field of view narrows a little more every cycle.

The positives that do land in the log are naturally thin. In Mailchimp’s cross-industry benchmarks, email averages a 2.62% click rate; Airship’s push-notification numbers put the median direct-open rate at just 7–8%; Criteo’s standard uplift dataset has a 0.292% conversion rate across 14 million rows. And the sample keeps moving: campaigns run cycle after cycle, and both who the audience is and how they respond drift over time. Didi’s marketing paper splits this drift in two, the feature distribution drifting and the feature-to-response relationship drifting.

Add it up and the sample has four problems. Biased: the training population and the scoring population don’t match. Narrow: the feedback loop tightens the model’s view of who exists, cycle by cycle. Thin: positives are scarce to begin with. Drifting: the distribution moves with the campaign calendar.

Biased is the easiest. The classic statistical fix is weighting. What you really want to minimize is expected loss over the full population, but your sample comes from the targeted population; multiply each sample by a weight equal to the ratio of its density under the full distribution to its density under the targeted one, and after weighting, the training loss equals the full-population loss in expectation. The idea traces to Shimodaira in 2000; in high dimensions both densities are hard to estimate, so later methods estimate the ratio directly. In operations there’s a shortcut: since targeting is done by your own rule, you can replay the historical logic, or train a “probability of being targeted” model on the targeting logs (the propensity score from the causal-inference literature), and weight each sample by the inverse of that probability, so the rarely-targeted samples count for the most. One step further is the doubly robust estimator, which Dudík, Langford, and Li brought into policy evaluation in 2011: build both a response model and a targeting-probability model, and as long as one of the two is right, the final estimate is right. This fails in two places: samples with near-zero targeting probability get explosive weights and explosive variance, so you truncate; and probability exactly zero is unrecoverable — no weight can conjure a label for people who were never targeted at all.

Narrow has no after-the-fact fix; you have to act at the source, by holding out a stream of random traffic. Random delivery cuts the link between who gets targeted and their features, so the samples that fall out carry no selection bias, and everyone’s probability of being targeted is known and non-zero, exactly the precondition weighting was missing. There are two ways to do it. One is pure random: carve out a fixed slice of campaign traffic and ignore the model score entirely. Alibaba’s entertainment arm does exactly this: samples touched by the algorithm don’t satisfy the conditions for causal identification and “can’t be used for training,” so they reserve “a small slice of randomly-probed traffic,” an unbiased sample “currently only on the order of a few million,” and train the uplift model on it alone. The other way doesn’t carve out a separate slice: the targeting policy shifts from a hard threshold to a logged probabilistic one, most traffic still goes by the model, but every decision carries a non-zero probability written into the log. Uber’s CRM outreach builds this in: LinUCB inside a campaign (pick the action by “predicted value plus an uncertainty bonus,” so the under-tried options automatically get more chances; its character is optimism), SquareCB across campaigns (assign each action a sampling probability from its predicted score and draw one at random; its character is randomization), probing as it delivers. Google went further and wrote its recommender’s exploration allocation as an optimization problem outright.

Pure random and probabilistic are both randomization; the difference is where the randomness is spread. Pure random concentrates it in one small slice, which becomes a clean randomized control sample good enough to train and evaluate an uplift model; the cost is that the rest of the log still carries no probabilities and can’t be reused. Probabilistic thins the randomness across all traffic, and in exchange every log line carries a probability, so any future offline evaluation of any new targeting policy can use the full data. Recruit’s coupon paper is about exactly this trade-off: a deterministic policy’s logs block future offline evaluation, so they mix the model policy and random exploration at a tunable ratio. As for how much randomness to spend (the slice fraction, or the exploration strength), there’s no standard answer: too little and you can’t accumulate positives; too much and short-term performance suffers, so you settle it with the business.

Thin is handled by negative downsampling plus post-hoc recalibration. Train directly on a fraction-of-a-percent positive rate and nearly every batch is negatives: slow, and the model won’t learn; drop some negatives to bring the ratio closer and it can. The cost is that changing the positive-to-negative ratio inflates the model’s output probabilities, so you have to scale them back by the sampling rate. The other route skips downsampling and manufactures positives instead: synthetic oversampling. SMOTE draws a line between nearby positives and takes a point on it as a new positive; a generative model like a VAE first learns the distribution of the positives and samples from it. The VAE looks more sophisticated than SMOTE’s straight-line interpolation, but in the tabular comparisons it often ties or slightly loses to SMOTE, and the more realistic the synthetic samples, the worse downstream classification gets. The reason is that anything built by changing the class ratio falls into the same trap: a 2022 systematic review in medical risk prediction found that random oversampling, undersampling, and SMOTE all badly overestimate minority-class probabilities, and the authors concluded that class imbalance itself isn’t the problem; correcting it can be what breaks the model. On top of which, at a fraction of a percent positive, the VAE is itself in the data-starved regime and won’t learn the positive distribution well. Whether to do synthetic sampling at all is a case-by-case call; often enough, the payoff is slight.

Drifting is the hardest to cure; the standard tool is frequent retraining. Retrain on a rolling recent window, let new samples keep pushing out old ones, and add calibration monitoring that fires a retrain when predictions drift systematically from actual conversion. Didi goes finer on its marketing stack: updating the causal-effect estimate incrementally, using knowledge distillation to keep what the old model already learned, and monitoring and handling feature-distribution drift and response-relationship drift separately. The idea is to accept that the distribution will move and let the model move with it. Retraining can still fall behind: big sales events and holidays shift both the audience mix and the conversion pattern at once, faster than a rolling window can follow, and you’re left either building a separate model for the occasion or accepting a few days of a model that’s off.

These four fixes don’t carry equal weight. Weighting, calibration, and retraining are all statistical patching, each with its own preconditions; the only one that solves the problem at the root is random traffic, because it changes how the data is produced. The cost is on the table too: a slice of delivery deliberately ignores the score and gives up some short-term performance, in exchange for a sample the model never touched.

Label: no response ≠ no interest

With the sample handled, the label is next. A response (binary) model has two labels, positive and negative. Most of the time, when a campaign ends, the converters are positives and the non-converters are negatives. But break the negatives apart and you find several kinds of customer: those the SMS or push never reached; those it reached but who never saw it, buried in the notification tray; those who saw it, were moved by it, but bought after the window closed or through another channel; and those who saw it and simply weren’t interested.

The first three are fundamentally different from the fourth. The first three were either not reached, not seen, or actually converted without being booked to this campaign; none of them says the person had no interest. Only the fourth — saw it, considered it, passed — is a true negative. Yet all four get filed as negatives without distinction. Machine learning has a name for this: PU learning (positive-unlabeled learning), where only the positives are reliable and everything else is unlabeled, positives and negatives mixed together.

Elkan and Noto proved in 2008 that as long as “which positives get labeled” is completely random and independent of user features, the model’s output ranking is correct. In marketing, that precondition simply doesn’t hold.

“Labeling completely at random” was later named SCAR in the literature. It fails in marketing because who gets reached and who gets to see the message depends everywhere on user features and can’t be truly random. Active users are more likely to catch a push; high-frequency customers are more likely to finish converting before the window closes. Whether a positive gets labeled depends on user features: the labeling probability is a function of the features. The PU literature calls this biased case SAR, the counterpart to the fully-random SCAR.

The other unavoidable quantity is the class prior, the true fraction of all users who would convert. You need it to turn the PU classifier’s scores back into real conversion probabilities; it also tells you roughly how many positives are hiding in the unlabeled pile, and how many people to target. But the class prior is itself hard to get: from PU data alone, it is mathematically unidentifiable. A shortage of labels can be explained by there genuinely being few positives, or by a low labeling rate, and the two explanations can’t be told apart from the data. SAR, and the unidentifiable prior, are the two places PU is genuinely hard.

So the first data step in a marketing setting lands on the logs. SMS has delivery receipts, push has impression records: pull the “never reached” people out of the unlabeled pile, treat them as missing data, and stop counting them as negatives; what’s left after that is the part that actually needs PU methods. This step is the data engineering that no marketing modeling gets to skip.

Then, in the modeling step, there are two traditional approaches.

The first is the two-step method, which started in text classification in 2002. Step one finds reliable negatives: pull a small fraction of the known positives out as “spies,” mix them into the unlabeled set, train a classifier to separate positives from the spy-laced unlabeled set; the spies are positives, so the scores they get tell you roughly where the real positives sit, and unlabeled points scoring below the spies can serve as reliable negatives. Step two trains an ordinary classifier on the original positives and the reliable negatives you pulled out. It’s a heuristic (pick the reliable negatives wrong and everything downstream goes wrong), but it’s intuitive and easy to debug.

The second is Elkan and Noto’s own weighting method: train a model that separates “labeled vs. unlabeled,” use its average score on the known positives to estimate that fixed constant, then use the constant to calibrate scores back to true probabilities. Its precondition is still SCAR; estimate the constant wrong and the whole thing tilts with it.

There’s a third family: risk rewriting. The idea is to assemble an unbiased estimate of the true classification risk out of “positives + unlabeled,” which is uPU. Write the class prior as π; the total risk splits into positive-class risk plus negative-class risk, where the negative-class risk can’t be computed directly and is estimated as the negative loss on the unlabeled samples minus the part of the positives counted as negative:

R̂_N = E_U[ℓ⁻] − π·E_P[ℓ⁻]

This term should be non-negative, but once the model is flexible enough to start overfitting, its empirical value gets pushed below zero, which is clearly wrong. nnPU in 2017 fixes it directly by putting a max(0, ·) around that term so it can’t drop under zero, and it became the common starting point for PU training. This family also needs the class prior as input, and since the prior is unidentifiable, it can only be estimated under extra assumptions: TIcE uses a decision tree to find an “almost pure” subspace and is fast; DEDPUL does cheap post-processing on any classifier’s output. The SAR family, meanwhile, converges with causal inference: since the labeling probability depends on features, model it, like the propensity score in selection bias, and correct the loss with inverse-probability weighting, which is the road PUe (NeurIPS 2023) takes.

PU has had unbroken academic output for over twenty years, yet the first systematic benchmark only appeared in 2025, its authors judging that experimental setups across papers were too inconsistent to compare. The public production cases cluster in risk control and fraud; for marketing coupons, first-hand public cases are close to nonexistent. The bigger blank is at the intersection: uplift modeling stacked on incomplete labels, two hard problems multiplied, and there’s still no systematic study of it today.

Target: optimizing a quantity you can’t observe

Sample and label done, back to the target. Look again at Operator 2: the real problem was never that the score was inaccurate. What the retention campaign wants isn’t “who will churn,” it’s “who will stay because of this call” — the increment the intervention creates. A customer’s score on those two questions is not the same.

Ascarza quantified this in a 2018 marketing paper: targeting by highest churn risk is not the most effective retention strategy; switch to targeting by “sensitivity to retention” and the same budget can cut churn by up to a further 7 percentage points. That earlier white paper had already drawn a four-quadrant picture, splitting people into four kinds: those who stay if treated (Persuadables), those who stay regardless (Sure Things), those who leave regardless (Lost Causes), and those an intervention actually drives away (Sleeping Dogs). A response model can’t tell these four apart: it just ranks Sure Things and high-risk Lost Causes up front, the former a wasted subsidy, the latter money down the drain, while Operator 2’s churn-raising call landed squarely on Sleeping Dogs.

So the target changes from P(Y|X) to τ(x) = E[Y(1) − Y(0) | X] — the difference in outcome for the same person treated versus not. The field calls this quantity the conditional average treatment effect (CATE), and the models that estimate it are collectively the uplift models.

Changing the target brings its own problem right away. Of the Y(1) and Y(0) in τ(x), you only ever see one for any given person: they either got the coupon or didn’t, and both can’t happen. The uplift label simply doesn’t exist. Gutierrez and Gérardy put their finger on it in a review: we never observe the true τ, which leaves standard supervised learning with nothing obvious to grab onto: supervised learning needs labels, and here you can’t manufacture even one true label.

The whole history of uplift modeling is a history of routing around “no label.” Radcliffe and Surry proposed differential response in 1999: use a control group to subtract off the “would have converted anyway” part and isolate the intervention’s net effect, the starting point. The most intuitive approach is the two-model method: train one model on the treatment group, one on the control, and subtract the predictions. Radcliffe pointed out its flaw long ago: both models are fitting “will they convert,” the main effect, and neither is fitting “the difference between the two” directly. And the increment is a second-order quantity, much weaker than the main effect, so subtracting two large numbers easily drowns a faint signal in noise. Hence trees that grow toward the increment directly (Rzepakowski and Jaroszewicz, 2010, swap the split criterion to “which cut spreads the two groups’ conversion distributions the most”); then the meta-learner family (S/T/X/R/DR, recasting the problem as a few standard regressions); causal forest, the one branch that also hands you a confidence interval (Wager and Athey, 2018); and the deep family, doing representation learning (TARNet, DragonNet, plus domestic coupon-targeting networks in production like Meituan’s ECUP and Kuaishou’s UniMVT).

Line these methods up for a large-scale comparison and the conclusion is a little counterintuitive. In a 2026 test on Criteo-scale data scored by Qini, first place goes to the plainest S-learner paired with a LightGBM, with every deep network behind it. Another benchmark spanning a dozen deep uplift models is blunter still: change the dataset and the best model changes, none holds the crown. The problem still traces back to the increment being a second-order quantity relative to the response: the signal is an order of magnitude weaker, and the more complex the model and the more parameters, the faster variance climbs and the easier it buries that faint signal. No single test settles it, but several independent results line up the same way, and the edge of deep models over the traditional ones is smaller than you’d expect.

The increment target brings one more problem: it needs both a treatment and a control group, and it’s demanding about how random those two groups are. Every one of these problems is really about how to correct the non-randomness of the two groups. So, once again, we loop back to random traffic.

Evaluation: an exam with no answer key

The last section built the uplift model; this one is about how to grade it. On other machine-learning tasks that’s a non-problem: hold out a test set, compare predictions to true labels, and there’s your score. But the uplift model has no true labels. Of the two outcomes in τ(x), you see only one per person: the model predicts a given customer’s increment at 3 percentage points, and you can never go back and check whether he actually stayed because of that call. Radcliffe, in a 2011 paper devoted to uplift evaluation, admitted that “surprisingly little work” has been done here, the bustle of modeling forever out of proportion with the quiet of evaluation. Evaluation is harder than modeling, and the hard part is the missing ground truth.

The existing approach sidesteps grading person by person and settles accounts at the group level. Rank everyone by predicted increment, high to low, then walk a randomized control sample in segments: among the top 10%, how much do treatment and control conversion differ; among the top 20%; on to the full population. A well-ranked model puts the high-increment people up front, so the treatment-control gap is widest at the head and flattens toward the tail. Connect the cumulative increment across segments and you get the uplift curve; the area it clears above the “random targeting” diagonal, normalized, is the Qini coefficient. In this scheme the randomness of the two groups is the key factor. You can’t get any individual’s counterfactual, so you lean on treatment and control being unbiased estimates of each other. But if the two groups are themselves biased, the unbiased estimate doesn’t hold.

The path looks straight, but it has plenty of traps, and every trap points to one thing: the metric is itself a measurement, and when the data isn’t clean, it lies.

First, Qini and the uplift curve have no single definition. Devriendt and colleagues surveyed this: the incremental expectation has “many different ways” to compute, and the literature “frequently lacks a formal description of the curve computation.” AUUC (Area Under the Uplift Curve) is a name that stuck by convention; every open-source tool calls its output AUUC, and they aren’t necessarily the same quantity, and the same model’s score can change when you switch libraries.

Second, these metrics only hold on randomized data. A Criteo evaluation-methods paper proves AUUC “fails on non-randomized control trial datasets” and should be used only for RCTs. Run Qini on observational logs the model already targeted and a number still comes out, it just doesn’t count.

Third, on biased data the metric doesn’t merely distort, it ranks models wrong. A 2026 paper on structural bias found that on biased data, proxy metrics like Qini and AUUC “give erroneous model rankings”: the “best” model you pick by them may not be the best at all, while metrics that are mathematically aligned with the ATE rank more stably when the data is imperfect.

Evaluation metrics have moved along recently too. To correct the bias in the evaluation set, our old friend the propensity score comes out again. Zhao and colleagues gave an “expected response” evaluation in 2017: use inverse-propensity weighting (IPW) to estimate, unbiasedly on randomized data, “how much conversion you’d get if you targeted by this model.” Verbeke and colleagues then brought a cost-benefit matrix in and computed incremental profit directly.

Beyond fixing AUUC and Qini, another route is to replace the metric outright. A common one is the R-loss: Nie and Wager estimate the main effect and the propensity first, then assemble an objective that “keeps only the causal part of the signal.” As for which proxy metric to trust, a large-scale test at ICLR 2024 compared 34 metrics across 78 datasets: there’s no universal one, but “metrics that incorporate doubly robust structure dominate across the board.” Doubly robust, as in the sample-bias fix earlier: an estimate carries two models, one predicting response and one predicting propensity, and as long as one is right, the result is unbiased. A metric wearing this structure doesn’t have to bet on perfectly clean data. Trusting doubly-robust proxy metrics for model selection is roughly the current consensus.

Academically there are still gaps: calibration of the increment, confidence intervals, and more. But in engineering a fixed workflow has settled: CausalML packages validation into a set routine, EconML breaks it out into a standalone DRTester. The standard industry posture in China has stabilized too: accumulate clean samples with random bucketing first, pick the model offline by Qini/AUUC, then run a real A/B control online as the final gate.

The lesson of this link is the same sentence as the others: settle the evaluation protocol before you talk about model selection; an offline metric only earns the right to speak once the bias is corrected. However pretty your Qini score, if the control and treatment groups are wildly biased, it’s just a number talking to itself.

Attribution: many campaigns, who gets the credit

The first four links all played out inside a single campaign. In reality campaigns never run alone: coupons, push, SMS, outbound, membership perks: a dozen live at once, one user gets brushed by three or four in a week, and finally places one order. Who gets the credit for that order is what attribution has to answer. It looks like bookkeeping, but it’s the hardest of the five links, because there’s no impartial referee to decide who the credit belongs to.

The most direct fault is double-counting. Clicked a search ad Monday, saw a retargeting ad Wednesday, bought Friday, and all three channels book that order into their own report. Someone on the ad-platform side did the reconciliation: platforms’ self-reported conversions summed to 30–50% more than actually occurred, and in one worked example the platforms collectively claimed 175 conversions against an actual 95, an 84% overcount. Inside a company it’s the same: every operations team’s ROI report looks good, and the increments added up don’t match real revenue.

A layer deeper, the credit that attribution records mixes in the “would have bought anyway” part. A user who’d order without a coupon still gets the credit assigned by last-touch to the last campaign he happened to brush. That doesn’t just misbook; it distorts incentives. Berman proved in 2018 that last-touch systematically over-incentivizes exposure and actually lowers advertiser profit, and there’s no condition under which it beats Shapley-value attribution. The party being measured will game the rule too: teams pile up coupons the night before a big sale to grab the last touch, and that’s the equilibrium.

So how to allocate credit correctly becomes the question. Multi-touch attribution (MTA), which weights each touchpoint by position, was long ago caught on a root flaw: it compares the conversion rate of “people who saw this touchpoint” against “similar people who didn’t,” two different groups of people, and can’t answer “would this same person have bought without this touchpoint”, which is exactly what decides whether the credit is deserved. User-level deep attribution tried carrying on. Alibaba and the CAS’s CausalMTA (2022) went after precisely this bias: heavy buyers see more ads and convert more readily anyway, so the touchpoints are riding on those people; it first uses a model (a variational autoencoder) to strip that “already a heavy buyer” preference out of the data, level the populations facing different touchpoints, then does Shapley allocation. It’s the capstone of that line, with almost nothing new at the top venues since. And causal attribution on purely observational data was judged unreliable by Gordon and colleagues across 663 controlled experiments: even with DML and five-thousand-plus user-level features, the estimated effects were off by multiples from the real experiments, the authors’ own line being that they were “unable to reliably estimate an ad campaign’s causal effect.”

Two families survive.

One is observational methods that know their own limits. Google’s data-driven attribution (DDA) borrows the Shapley value from game theory: treat a conversion as jointly produced by several touchpoints, ask one by one “how much does the conversion probability drop if we remove this touchpoint,” average over the various orderings, and split credit by each touchpoint’s marginal contribution. It’s a fair split, but the docs say it plainly: that “what if we removed this touchpoint” is estimated from similar users’ conversion probabilities, not a counterfactual from a real experiment. When the definition is consistent and you only want relative ranking across campaigns, it’s enough; just don’t take its absolute numbers as true increments.

The other is the honest incrementality-experiment toolbox, all of which need a control group. The cleanest is a global control: hold out a small set of users who never get touched, and use them as the baseline. On the ad side there’s ghost ads, which log “who would have seen the ad” instead of serving the control group real PSAs, saving the cost of a PSA test. Under algorithmic delivery, though, the treatment and PSA populations diverge, and the older PSA-based control breaks down. When user-level control isn’t available, drop to geo: pair cities at random, treat half and not the other (Google’s Trimmed Match and Meta’s GeoLift, at the cost of giving only campaign-level totals). CRM has a standard correction called triggered analysis, which counts only the users actually triggered, though a 10% effect measured on the triggered group doesn’t mean the full population moves 10%.

Assemble all this into a company-wide measurement standard and you get tiered holdout: an outermost global control that’s never touched, then program-level and campaign-level controls inside it, each answering its own increment. What makes it valuable is that single campaigns’ effects summed together overstate, and only a global control can measure the true cumulative increment after all campaigns stack.

The newest route replaces observational attribution wholesale, extrapolating from real experiments instead. Gordon and colleagues’ PIE works like this: gather a batch of campaigns that were run as proper A/B tests and whose true increments are known; train a model to learn “what kind of campaign (which audience, which channel, which offer) tends to produce how much increment”; then for a new campaign that wasn’t experimented on, feed its features in and predict the increment directly, skipping a fresh experiment. To score how accurate it is, you measure how well the predicted increments match what the real experiments measured, using R² (1 is a perfect match, 0 is no predictive power): PIE reaches 0.88, while the industry-standard 7-day last-click attribution manages only 0.19, barely recovering the true increment at all. It’s “experiment-grounded, model-amplified,” which amounts to conceding that the pure-observational road is a dead end and only experiment can be the foundation.

Beyond the horizontal problem of many campaigns, there’s a trap in time. A good chunk of a promotion’s “increment” is just future purchases pulled forward: with a coupon in hand, people buy earlier and stock up more, and once the promotion ends, sales dip. Store-level data puts the combined pre- and post-promotion dip at 4–25% of the current-period increment; another study decomposing the promotional bump found only 33% is genuine brand switching, the rest being pull-forward and stockpiling. So even a well-measured campaign-level increment has to be looked at again over a longer horizon, otherwise part of the net lift you think you have was only borrowed from next month.

This link’s conclusion leans on organization more than any before it: the one measurement standard that holds up inside a company is a tiered holdout. Who gets the credit comes down, in the end, to whether you’re willing to hold out that never-touched control group — not to a cleverer allocation formula.

What if you have no random group

Every link’s conclusion has landed on random traffic. Which raises the unavoidable counter-question: is a random group really necessary? Plenty of teams don’t keep one, or can’t afford to: carve out a slice that ignores the model and short-term numbers look bad, and the business is the first to object. So this section answers it head-on: without a random group, how far can you get, and at what cost.

First, one distinction: training an uplift model and evaluating it don’t depend on randomness equally. Training has some room to maneuver; evaluation has almost none, because Qini and AUUC only stop lying on randomized data. So the routes below can mostly help you train a model, but won’t necessarily validate it for you.

Route one: find the randomness in past delivery. Even without a dedicated random group, as long as the old targeting policy carried a little randomness, with each person’s probability of being targeted known and non-zero, you can use propensity weighting to back out the effect of policies you never actually ran, which is off-policy evaluation (OPE). In marketing operations the propensity is usually free: the targeting rule is your own, so replaying the history gives it to you, no guessing. The worst case is hard-threshold targeting: over the line, target; under it, don’t — everyone’s probability is 0 or 1, the two groups don’t overlap at all, and propensity weighting fails outright. Recruit made this point about coupons: a deterministic policy blocks future offline evaluation.

Route two: borrow the data’s natural structure. Targeting by a score threshold builds a discontinuity by itself: people who land just on either side of the threshold have nearly identical features, one targeted and one not, exactly where regression discontinuity (RDD) shines. Hartmann and colleagues argued back in 2011 that firms targeting by score or threshold are a natural fit for RDD. The cost is plain: RDD only gives a local effect for the sliver near the threshold, geo experiments only give a campaign-level total, and neither buys you a full-population user-level ranking.

Get Zhonghm’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Route three: bulldoze the confounding. Assume you’ve measured all the confounders that need controlling (causal inference calls this ignorability) and you can estimate causal effects on observational data directly: propensity matching, IPW, double machine learning (DML), DR-learner, causal forest; the whole toolbox is here. But that assumption often fails in marketing, which is exactly what Gordon’s 663 experiments already proved. How far this route goes depends on whether you believe “everything that needed measuring was measured,” and that’s usually not something you can prove yourself.

Route four: change the grain, change the paradigm. Since individual-level randomness is out of reach, stop fighting at the individual level: marketing mix modeling (MMM) retreats to aggregate data and estimates elasticities from total spend versus total sales, needing no individual randomness at all; PIE’s experiment-library extrapolation belongs here too. The cost, again, is giving up the user-level ranking.

Looking further out, the direction drawing the most heat lately is stronger representations, even foundation models, to push the old “we didn’t measure all the confounders” problem forward. One branch is balanced representation (TARNet, CFR let treatment and control share a learned representation, forcing the two distributions into alignment before estimating). One is a transformer encoding a user’s long behavior sequence into a general user representation, in the hope it absorbs into features the confounders normally too diffuse to measure. The most aggressive is the causal foundation model: CausalPFN brings in-context learning to causal estimation, pretraining once on a mass of simulated data satisfying ignorability, then mapping observations to causal effects directly at inference with no task-specific tuning, already competitive on uplift tasks.

But however well the representation learns and however big the encoder, all it improves is “how accurately you estimate,” not “whether you can identify.” These are two different things. Identification asks whether the causal effect can be separated out of the data at all, which turns on whether the factors affecting “whether to target” and “whether they convert” were all recorded (that’s ignorability); estimation asks, given the effect is separable in principle, how accurately you can compute it from the sample in hand. A stronger model helps only with the second. A confounder that never entered the data can’t be conjured by a bigger encoder. And to this day, “pretrain a general user representation, then explicitly use it for uplift” is a road no one has yet connected: existing general user representations feed recommendation, CTR, and retention downstream, and not one lands on the increment. The direction is right, but there’s no mature, drop-in method yet.

To wrap up: you can go without a random group, but you can’t go without some known source of randomness or exogenous variation: borrow the randomness left in a past policy, borrow the natural structure of a threshold or a geography, or retreat to an aggregate grain. Each of the four routes has its own blind spot, and none hands you clean, full-population, user-level increment for free. So the genuinely economical move is to make randomness thin and cheap: turn targeting from an over-the-line hard threshold into a logged probabilistic policy: small cost, and it leaves a door open for every future offline evaluation. As for stronger representations and foundation models, they make estimation cheaper, but the identification hurdle doesn’t vanish just because the model got bigger. All the way around the loop, randomization is still the foundation you can’t skip.

Fix the measurement before the model

Walk through all five links and you find the fixes are five faces of the same thing. The sample link is short a stream of randomly-probed traffic; the label link needs reach and impression recorded faithfully in the logs; the target link needs a randomized experiment to manufacture an increment label; evaluation needs a randomized test set; attribution needs a full tiered-holdout system. Round and round, it all loops back to randomization and the measurement built on it. In marketing modeling, randomization and measurement infrastructure are the first-order investment, and the model is second-order.

The order can’t be reversed. As long as the measurement chain is bad, dropping a stronger model on top of it just makes the wrong answer converge faster, ranking Sure Things to the front more crisply, delivering subsidies to the would-buy-anyway crowd more precisely. Operator 2’s churn-raising call: the churn model itself was fine; what was wrong was the whole chain measuring by response instead of by increment.

The new LLM-era tools don’t rewrite this conclusion. A causal foundation model automates the estimation and leaves the identification hurdle untouched for the user; LLM-generated synthetic consumers, standing in for real survey respondents, reach 90% of human test-retest reliability at the group level but fail completely at the individual level, good only for a rough pre-campaign screen, no substitute for an incrementality experiment. The tools keep getting stronger, but what they save is the computing — they can’t save you from building a dataset that can identify cause.

Two open questions to close on. One is the intersection of incomplete labels and uplift: PU stacked on uplift, two hard problems multiplied, still with no systematic study. The other is more basic: an offline Qini that leads while the online lift can’t be measured is something almost everyone has hit, yet a named public empirical account of it still doesn’t exist.

A lap through all this, and I’m more sure than before: the hardest part of marketing modeling was never picking the algorithm. The algorithms are off the shelf, open-source, written out plainly in the papers; what’s genuinely hard, and genuinely valuable, is whether you’re willing to pay for knowing the truth — holding out that never-touched control group, carving out that short-term-ugly stream of random traffic.

Suggested tags: Uplift Modeling · Causal Inference · Marketing Analytics · Machine Learning · Incrementality Testing

References

1. Radcliffe & Simpson, Identifying who can be saved and who will be driven away by retention activity, Stochastic Solutions white paper, 2007. https://stochasticsolutions.com/pdf/SavedAndDrivenAway.pdf
2. Zadrozny, Learning and evaluating classifiers under sample selection bias, ICML 2004. https://dl.acm.org/doi/10.1145/1015330.1015425
3. Swaminathan & Joachims, Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization, JMLR 2015. https://jmlr.org/papers/v16/swaminathan15a.html
4. Mailchimp, Email Marketing Benchmarks & Industry Statistics (2023–12). https://mailchimp.com/resources/email-marketing-benchmarks/
5. Airship, Mobile App Push Notification Benchmarks for 2025. https://growth.airship.com/rs/313-QPJ-195/images/Airship-2025-Push-Notification-Benchmarks-EN.pdf
6. Criteo AI Lab, Criteo Uplift Prediction Dataset (Diemert et al., AdKDD 2018). https://ailab.criteo.com/criteo-uplift-prediction-dataset/
7. Didi, The Estimation of Continual Causal Effect for Dataset Shifting Streams, arXiv:2504.20471, 2025. https://arxiv.org/abs/2504.20471
8. Shimodaira, Improving predictive inference under covariate shift by weighting the log-likelihood function, JSPI, 2000.
9. Sugiyama et al., Direct Importance Estimation (KLIEP), NIPS 2007. https://proceedings.neurips.cc/paper/2007/file/be83ab3ecd0db773eb2dc1b0a17836a1-Paper.pdf
10. Dudík, Langford & Li, Doubly Robust Policy Evaluation and Learning, ICML 2011. https://arxiv.org/abs/1103.4601
11. Alibaba Entertainment, Smart marketing uplift-model practice (DataFunTalk), 2020. https://cloud.tencent.com/developer/article/1620903
12. Google, Item Level Exploration Traffic Allocation in Large-scale Recommendation Systems, arXiv:2505.09033. https://arxiv.org/html/2505.09033v1
13. Uber, Enhancing Personalized CRM Communication with Contextual Bandit Strategies, 2025. https://www.uber.com/blog/enhancing-personalized-crm/
14. van den Goorbergh et al., The harm of class imbalance corrections for risk prediction models, JAMIA 29(9), 2022. https://academic.oup.com/jamia/article/29/9/1525/6605096
15. Nishimura et al., Balancing Immediate Revenue and Future Off-Policy Evaluation in Coupon Allocation, arXiv:2407.11039, 2024. https://arxiv.org/abs/2407.11039
16. Elkan & Noto, Learning Classifiers from Only Positive and Unlabeled Data, KDD 2008. https://cseweb.ucsd.edu/~elkan/posonly.pdf
17. Bekker & Davis, Learning from positive and unlabeled data: a survey, Machine Learning 109, 2020. https://arxiv.org/abs/1811.04820
18. Bekker, Robberechts & Davis, Beyond the SCAR Assumption for Learning from Positive and Unlabeled Data, ECML PKDD 2019. https://link.springer.com/chapter/10.1007/978-3-030-46147-8_5
19. Liu, Lee, Yu & Li, Partially Supervised Classification of Text Documents (S-EM), ICML 2002. https://www.cs.uic.edu/~liub/S-EM/unlabelled.pdf
20. PU Learning in risk control (practitioner blog), ~2019. https://zhuanlan.zhihu.com/p/98887617
21. Kiryo et al., Positive-Unlabeled Learning with Non-Negative Risk Estimator, NeurIPS 2017. https://arxiv.org/abs/1703.00593
22. Bekker & Davis, Estimating the Class Prior in Positive and Unlabeled Data through Decision Tree Induction (TIcE), AAAI 2018. https://cdn.aaai.org/ojs/11715/11715-13-15243-1-2-20201228.pdf
23. Ivanov, DEDPUL, ICMLA 2020. https://arxiv.org/abs/1902.06965
24. Wang et al., PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference, NeurIPS 2023. https://openreview.net/pdf?id=6vtZIoxZoJ
25. Verifying the SCAR Assumption in Positive-Unlabeled Learning, arXiv:2404.00145, 2024. https://arxiv.org/abs/2404.00145
26. Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms, arXiv:2509.24228, 2025. https://arxiv.org/abs/2509.24228
27. Soft Label PU Learning (Tencent), arXiv:2405.01990, 2024. https://arxiv.org/html/2405.01990v1
28. Minority Class Oversampling for Tabular Data with Deep Generative Models, arXiv:2005.03773, 2020. https://ar5iv.arxiv.org/html/2005.03773
29. Synthetic Data Augmentation for Imbalanced Tabular Data: A Comparative Study, MDPI Electronics, 2026. https://www.mdpi.com/2079-9292/15/4/883
30. Ascarza, Retention Futility: Targeting High-Risk Customers Might Be Ineffective, JMR 55(1), 2018. https://journals.sagepub.com/doi/abs/10.1509/jmr.16.0163
31. Gutierrez & Gérardy, Causal Inference and Uplift Modelling: A Review, PMLR v67, 2017. https://proceedings.mlr.press/v67/gutierrez17a.html
32. Radcliffe & Surry, Differential Response Analysis, Credit Scoring and Credit Control, 1999.
33. Rzepakowski & Jaroszewicz, Decision trees for uplift modeling, ICDM 2010. https://home.ipipan.waw.pl/sj/pdf/ICDM10web.pdf
34. Künzel et al., Metalearners for estimating heterogeneous treatment effects, PNAS 116(10), 2019. https://www.pnas.org/doi/10.1073/pnas.1804597116
35. Wager & Athey, Estimation and Inference of Heterogeneous Treatment Effects using Random Forests, JASA 2018. https://arxiv.org/abs/1510.04342
36. Shalit, Johansson & Sontag, Estimating individual treatment effect (TARNet/CFR), ICML 2017. https://arxiv.org/abs/1606.03976
37. Shi, Blei & Veitch, Adapting Neural Networks for the Estimation of Treatment Effects (DragonNet), NeurIPS 2019. https://arxiv.org/abs/1906.02120
38. Meituan, ECUP: Entire Chain Uplift Modeling with Context-Enhanced Learning, arXiv:2402.03379, 2024. https://arxiv.org/abs/2402.03379
39. Kuaishou, UniMVT: Jointly Optimizing Debiased CTR and Uplift for Coupons Marketing, arXiv:2602.12972, 2026. https://arxiv.org/html/2602.12972
40. A Large-Scale Empirical Comparison of Meta-Learners and Causal Forests for HTE in Marketing Uplift, arXiv:2604.06123, 2026. https://arxiv.org/html/2604.06123
41. Benchmarking for Deep Uplift Modeling in Online Marketing, arXiv:2406.00335, 2024. https://arxiv.org/html/2406.00335v1
42. Radcliffe, Quality Measures for Uplift Models, Stochastic Solutions (KDD 2011 workshop), 2011. https://stochasticsolutions.com/kdd2011late.html
43. Devriendt, Berrevoets & Verbeke, Learning to Rank for Uplift Modeling, arXiv:2002.05897, 2020 (IEEE TKDE 2022). https://arxiv.org/pdf/2002.05897
44. About Evaluation Metrics for Contextual Uplift Modeling, arXiv:2107.00537, 2021. https://arxiv.org/abs/2107.00537
45. Evaluating Uplift Modeling under Structural Biases, arXiv:2603.20775, 2026. https://arxiv.org/html/2603.20775
46. Zhao, Fang & Simchi-Levi, Uplift Modeling with Multiple Treatments and General Response Types, SDM 2017, arXiv:1705.08492. https://arxiv.org/abs/1705.08492
47. Verbeke et al., The Foundations of Cost-Sensitive Causal Classification, arXiv:2007.12582, 2020. https://arxiv.org/pdf/2007.12582
48. Nie & Wager, Quasi-Oracle Estimation of Heterogeneous Treatment Effects (R-learner), arXiv:1712.04912, 2017 (Biometrika 2021). https://arxiv.org/abs/1712.04912
49. Mahajan, Mitliagkas, Neal & Syrgkanis, Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation, ICLR 2024, arXiv:2211.01939. https://arxiv.org/abs/2211.01939
50. CausalML Validation docs. https://causalml.readthedocs.io/en/latest/validation.html
51. EconML Validation (DRTester) docs. https://www.pywhy.org/EconML/spec/validation.html
52. Ruler Analytics, How Double-Counting Conversions in Ad Platforms Skews Your Budget Allocation, 2026. https://www.ruleranalytics.com/blog/reporting/conversion-duplication/
53. Berman, Beyond the Last Touch: Attribution in Online Advertising, Marketing Science, 2018. https://ron-berman.com/papers/attribution.pdf
54. Google, MCF Data-Driven Attribution Methodology / Ads Data Hub Shapley Value (docs). https://support.google.com/analytics/answer/3191594
55. Yao et al. (Alibaba + CAS), CausalMTA: Eliminating Confounding Bias for Causal Multi-touch Attribution, KDD 2022, arXiv:2201.00689. https://arxiv.org/pdf/2201.00689
56. Gordon, Moakler & Zettelmeyer, Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement, arXiv:2201.07055 (Marketing Science 2023). https://arxiv.org/abs/2201.07055
57. Johnson, Lewis & Nubbemeyer, Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness, JMR 2017. https://journals.sagepub.com/doi/10.1509/jmr.15.0297
58. Zhao et al. (Google), Trimmed Match, arXiv:1908.02922 (AoAS 2022). https://arxiv.org/abs/1908.02922
59. Meta, GeoLift methodology docs. https://facebookincubator.github.io/GeoLift/docs/Methodology/ComparisonOthers/
60. Deng & Hu, Diluted Treatment Effect Estimation for Trigger Analysis in Online Controlled Experiments, WSDM 2015. https://dl.acm.org/doi/10.1145/2684822.2685307
61. Disney Streaming, Universal Holdout Groups at Disney Streaming, 2021. https://medium.com/disney-streaming/universal-holdout-groups-at-disney-streaming-2043360def4f
62. Uber, Building Uber’s Experimentation Platform (universal holdout), 2018. https://www.uber.com/us/en/blog/xp/
63. Gordon, Moakler & Zettelmeyer, Predictive Incrementality by Experimentation (PIE), arXiv:2304.06828, 2023. https://arxiv.org/pdf/2304.06828
64. Van Heerde, Leeflang & Wittink, The Estimation of Pre- and Postpromotion Dips with Store-Level Scanner Data, JMR 2000. https://journals.sagepub.com/doi/10.1509/jmkr.37.3.383.18782
65. Van Heerde, Gupta & Wittink, Is 75% of the Sales Promotion Bump Due to Brand Switching? No, Only 33% Is, JMR 2003. https://journals.sagepub.com/doi/abs/10.1509/jmkr.40.4.481.19386
66. Hartmann, Nair & Narayanan, Identifying Causal Marketing-Mix Effects Using a Regression Discontinuity Design, Marketing Science 30(6), 2011. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1525113
67. Pinterest, PinnerFormer: Sequence Modeling for User Representation, KDD 2022, arXiv:2205.04507. https://arxiv.org/abs/2205.04507
68. CausalPFN: In-Context Causal Effect Estimation, arXiv:2506.07918, 2025 (NeurIPS 2025). https://arxiv.org/abs/2506.07918
69. PyMC Labs × Colgate-Palmolive, Semantic Similarity Rating (SSR), arXiv:2510.08338, 2025. https://arxiv.org/abs/2510.08338
70. When Synthetic Users Fail, arXiv:2607.26348, 2026. https://arxiv.org/abs/2607.26348