Causal inference for commerce decisions: a design for treatment-effect estimation

1Context and motivation

Commerce teams make a steady stream of decisions about interventions: whether to offer free shipping, whether to remove a checkout step, which products to recommend, whether to show a payment-retry prompt. Most of the evidence they use comes from dashboards, funnel reports and segment comparisons. These describe what happened alongside an intervention. They do not say what the intervention changed.

The gap matters because the conditions around an intervention are rarely held still. A promotion launches in the same week as a traffic campaign; a new payment method becomes available in one region while a carrier changes its rates; a checkout change ships to the devices that already convert most often. In each case a naive before-and-after or treated-versus-untreated comparison mixes the effect of the intervention with the effect of everything that moved with it.

This paper describes the pipeline we propose for estimating the causal effect of such interventions in a commerce setting. We describe the design and the reasons for each part of it, drawing on the published literature in statistics, econometrics and machine learning. We do not report its effects, because we have not measured them.

2Foundations

2.1Potential outcomes

The design uses the potential-outcomes framework. For each unit (a session, a shopper or a store) and each treatment, there is an outcome that would occur under the treatment and one that would occur without it; the causal effect is the difference between the two¹. Only one of the two is ever observed for a given unit, which is the fundamental problem of causal inference, and every method in this paper is a way of filling in the missing one under stated assumptions².

Randomisation is the cleanest way to fill it in, because random assignment makes treated and untreated groups comparable in expectation. Where randomisation is not possible, carefully controlled non-randomised data can still be used to estimate causal effects, but only under further assumptions^{1, 3}.

2.2Causal graphs

Every decision the pipeline supports has a written causal graph. The graph states which variables are thought to cause which, which confounders must be adjusted for, and which assumptions cannot be tested from data alone⁴. We treat the graph as a reviewed artefact, owned by the team that owns the decision, rather than as something inferred automatically.

2.3Average and heterogeneous effects

The average treatment effect answers whether an intervention helps on average. Many commerce decisions need more: whether it helps the shoppers it would be shown to, and whether some groups are harmed. The conditional average treatment effect, the expected effect given a unit's observed characteristics, answers that question. Estimating it is harder than prediction because no unit's individual effect is ever observed, so the usual cross-validation against observed outcomes does not apply directly⁵. Uplift modelling, as it is called in marketing, addresses the same estimand; it has its own literature of tree-based and meta-learning methods that overlaps with the causal-inference one⁶.

3Design principles

The design follows six principles.

• Experiment first. If an intervention can be randomised at reasonable cost, it is randomised, and the experiment is the estimate of record⁷.
• Assumptions are written down. An observational estimate is published only with its causal graph and the assumptions it relies on⁴.
• Estimators are doubly protected. Observational estimates use estimators that remain consistent if either the treatment model or the outcome model is correct⁸.
• Heterogeneity comes with uncertainty. Effects per group are reported with confidence intervals, not as point scores⁹.
• Overlap is checked, not assumed. Units that had almost no chance of receiving the treatment, or almost no chance of not receiving it, are excluded from the estimate and reported as excluded³.
• Privacy is a design input. Released aggregates carry calibrated noise where the audience is broad¹⁰.

4Architecture

The pipeline has five components, plus a review step before any estimate is used.

4.1Data inputs

The pipeline reads the same event stream as the rest of the analytics stack: exposure to an intervention (which variant, offer or prompt a unit saw), the outcomes of interest (conversion, order value, return, repeat purchase at fixed horizons), and the covariates that the causal graph names as confounders. Typical confounders in this setting are device class, traffic source, region, prior purchase history, which payment methods were available, and whether shipping to the shopper's address was possible. Each record carries the timestamp of exposure so that covariates are taken from before treatment and outcomes from after it.

4.2Treatment assignment model

For observational estimates, the pipeline fits a model of the probability that a unit received the treatment given its covariates. Adjusting for this single score is sufficient to remove the bias from all the observed covariates it is built from, which allows matching, subclassification or weighting on one number instead of many¹¹. The model class is a design choice; we use a flexible learner and check calibration and the overlap of scores between treated and untreated units before any estimate is computed.

4.3Effect estimators

Three families of estimator sit in this layer.

• Doubly robust estimation of average effects. The estimator combines the treatment model with a model of the outcome; it is consistent if either is correctly specified, which gives two chances at a valid inference rather than one⁸.
• Double machine learning. When the set of confounders is large, flexible learners fit the nuisance models and the target effect is estimated with orthogonal moments and cross-fitting, which removes the regularisation bias that plugging machine-learning predictions into a regression would otherwise carry¹².
• Heterogeneous effects. Causal trees with honest sample splitting give valid inference on effects within data-driven subgroups⁵. Causal forests extend this to an estimate for each unit, with pointwise consistency and asymptotically normal sampling distributions, so confidence intervals can be reported⁹. Meta-learners, including the X-learner, build effect estimators from any supervised learner and adapt to structure such as a smooth or sparse effect; the X-learner is suited to the common commerce case where far fewer units are treated than untreated¹³.

4.4Counterfactuals for region-wide changes

Some interventions cannot be randomised at the unit level: a new payment method enabled for a whole country, a change in tax treatment, a carrier switch. For these the pipeline builds a counterfactual time series of what the outcome would have been without the change, from untreated regions and pre-period data, using a Bayesian structural time-series model. That approach estimates how the effect evolves over time and gives posterior intervals for it¹⁴.

4.5Serving and experimentation

Effect estimates are exposed to decision systems as versioned artefacts: an estimate, its interval, the causal graph and estimator that produced it, and the data window. A decision system that wants to target an intervention by estimated effect reads these artefacts; it does not retrain them. New targeting policies go through a randomised holdout before they replace an existing policy, so the experiment remains the final check on the observational work⁷.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Potential-outcomes definitions per decision	"Effect" used loosely for association	1, 2
Written causal graph	Unstated or untestable confounding assumptions	4
Randomised experiments where feasible	Confounding in observational comparisons	7
Propensity score with overlap check	Many observed confounders; units with no comparable controls	11, 3
Doubly robust estimator	Misspecified treatment or outcome model	8
Double machine learning with cross-fitting	Regularisation bias from high-dimensional nuisance models	12
Honest causal trees and causal forests	Average effects hide groups that are harmed or not helped	5, 9
X-learner for imbalanced treatment	Few treated units relative to controls	13
Bayesian structural time-series counterfactual	No unit-level randomisation for region-wide changes	14
Calibrated noise on released aggregates	Re-identification from published statistics	10

Source: our design, with the works cited in the table.

5Identification by intervention

The interventions in the original scope differ in how their effects can be identified. Table 2 assigns each to a strategy. It is a worked design table, not data: it records the choice we would make and why, and every row would need to be checked against the actual assignment process in a deployment.

Intervention	Unit of assignment	Primary strategy	Fallback when randomisation is not possible	Main confounder to adjust for
Free-shipping threshold	Session or shopper	Randomised experiment	Doubly robust estimate with propensity overlap check	Basket value before exposure
Removing a checkout step	Session	Randomised experiment	None; ship only behind an experiment	Device class
Price ending change	Product and region	Randomised experiment by product	Structural time-series counterfactual	Concurrent promotions
Recommendation strategy	Shopper	Randomised experiment with heterogeneous effects	Double machine learning on logged exposure	Prior engagement
Payment-retry prompt	Failed payment attempt	Randomised experiment	Doubly robust estimate	Decline reason and payment method

Source: our design.

Two patterns follow from the table. First, every row has a randomised primary strategy, because each of these interventions can in principle be assigned at random at the session, shopper or product level⁷. Second, the fallbacks differ by the structure of the problem: an intervention applied to a whole region needs a time-series counterfactual¹⁴, whereas one applied to individual shoppers with logged exposure can use unit-level adjustment^{8, 12}.

6Illustrative scenarios

The scenarios below are hypothetical. They show how the components would interact; they are not results.

Free shipping where shipping costs bite. A store that offered free shipping only to shoppers in regions with high delivery costs would see higher conversion there, but those regions might also differ in device mix and traffic source. An experiment within each region would separate the two. If the store also wanted to know which shoppers respond, a causal forest on the experiment's data would estimate the effect by basket value and region, with an interval for each⁹.

A payment method enabled for one country. If a new local payment method were enabled nationwide on one date, no shopper-level control group would exist. The pipeline would build the counterfactual conversion series from comparable countries and the pre-period, and report the estimated effect with its posterior interval over the weeks after launch¹⁴. A concurrent national holiday in that country only would break the comparison, and the causal graph would have to say so.

Regulation as a confounder. Strong customer authentication rules change checkout friction in some jurisdictions. A checkout simplification tested across jurisdictions would therefore have different effects by jurisdiction partly because of the regulatory step, not because shoppers there respond differently. The causal graph would carry the regulatory regime as a variable and the effect would be estimated within each regime⁴.

7Evaluation before any effect is reported

Before the pipeline's estimates could be used to target interventions, a deployment would need to pass the checks below.

• Overlap and balance. Propensity distributions for treated and untreated units overlap, and covariates are balanced after weighting or matching^{11, 3}.
• Agreement with experiments. Where both an experiment and an observational estimate exist for the same intervention, the two are compared; a persistent gap is evidence of unmeasured confounding.
• Placebo tests. The estimator is run on a period or outcome where the intervention cannot have had an effect, and should find none.
• Out-of-sample heterogeneity checks. Subgroup effects found in one sample are re-estimated in a held-out sample before they are used for targeting⁵.
• Targeting validated by holdout. A policy that targets by estimated effect is compared with random targeting in a controlled experiment⁷.

8Ethics, fairness and privacy

Targeting by estimated effect allocates interventions unevenly by design. The pipeline reports estimated effects by device class and region so that an operator can see whether a policy withholds a benefit, such as a shipping offer, from groups that would also gain from it. Demographic attributes are not used as model inputs. Aggregates released outside the analytics team carry noise calibrated to the sensitivity of the statistic¹⁰, and small cells are suppressed.

9Limitations

• No measurements. We have not measured any effect with this pipeline. Every benefit implied by the design is a hypothesis to be tested with the checks above.
• Unmeasured confounding. Observational estimates are only as good as the causal graph and the covariates recorded. Confounders that are not observed remain, and no estimator in this design removes them^{4, 3}.
• Overlap. Where treatment assignment is nearly deterministic, for example a feature shown only on one platform, there are no comparable controls and no observational estimate is possible.
• Interference. The potential-outcomes setup assumes one unit's treatment does not change another's outcome. Shared inventory, referral and price comparison can break that assumption in commerce.
• Small stores. Stores with little traffic or few distinct treatments will not have enough data for heterogeneous effect estimates; they would see average effects at most.
• Non-stationarity. An effect estimated in one season or market condition may not hold in the next. Structural time-series counterfactuals also degrade when the control series are themselves hit by a shock.
• Operator use. How operators read intervals and act on subgroup estimates is a human-factors question this design does not answer.

10Conclusion

We have described a pipeline for estimating the causal effect of commerce interventions: definitions in potential outcomes, a written causal graph for each decision, randomised experiments as the default, doubly robust and double machine-learning estimators where experiments are not possible, causal forests and meta-learners for heterogeneous effects, and structural time-series counterfactuals for region-wide changes. Each part rests on published methods, summarised in Table 1. The design has no evidence yet of its own accuracy. The next step is to run it on an intervention that also has a randomised experiment, and compare the two.