An analytics-to-action pipeline: from event telemetry to governed decisions

1Context and motivation

A dashboard describes the past. Turning what it shows into a change in what a store does next usually passes through a meeting, a ticket and a manual release, and each step adds delay. Teams also tend to optimise one channel, product line or region at a time, without the shared constraints (stock, margin, fairness, latency) that a global decision would respect. When business-intelligence models and machine-learning models are built from different copies of the data, their numbers drift apart and arguments about attribution follow.

We treat analytics as a control loop rather than a report: sense, model, decide, act, learn, and keep an auditable record of each step. This paper describes the architecture we propose for that loop in a commerce setting. The decisions in scope are product ranking, offers and markdowns, delivery promises, and proactive support outreach. We describe the design and the reasons for each part of it; we do not report its effects, because we have not measured them.

2Design principles

The design follows seven principles.

• Event first. Every meaningful state change is published as an event; batch processing is an optimisation for consolidation, not the default path.
• Causal before correlational. Where a decision allocates exposure, we prefer estimators of treatment effect over predictive association, and we make the identifying assumptions explicit in a causal graph^{1, 2}.
• Experiments carry guardrails. Every experiment declares guardrail metrics alongside its target metric and stops when a guardrail is breached^{3, 4}.
• Layers are separate but bound by contracts. The data foundation, feature layer, decision layer and action layer evolve independently, and each boundary is a versioned schema.
• Every decision is traceable. A decision can be traced back to the model version, policy version and feature snapshot that produced it.
• Budgets are explicit. Latency, infrastructure cost and model complexity each have a budget that a change must fit inside.
• People keep the high-impact calls. Decisions above a set impact level are reviewed by an operator, with a local explanation of the model's output⁵.

3Architecture

The pipeline has six layers, plus an observability and governance plane that spans all of them.

3.1Ingest and data contracts

Producers publish events against versioned schemas held in a registry. A payload that does not match its schema is rejected and the producer is quarantined until it conforms; contract tests run in each producer's build. Each event carries consent and residency flags, and the router writes it only to storage in its region. Critical fields (price, quantity, timestamps) have enum and range checks.

Raw events are stored immutably. Every derived table, feature and model input is a function of that raw record, so any of them can be recomputed when logic changes or a bug is found, and lineage from a feature back to its source events is a property of the system rather than a separate document⁶.

3.2Streaming and batch

Decisions read from a streaming path; long-horizon training, backfills and audits read from batch tables. Stream processing has to cope with duplicates and with events that arrive after the window they belong to. The design uses deduplication keys so that reprocessing is idempotent, and event-time watermarks to decide when a window is complete and how late data is reconciled⁷. Backfills cannot overwrite existing tables without lineage tags and a diff report.

3.3Canonical data model and feature platform

Conformed entities (user, session, product, offer, order, fulfilment node) give BI and ML one shared definition of each business object. The feature platform computes each feature with the same code online and offline, and parity is checked by comparing checksums and distributions between the two paths. Each feature has an owner, a refresh cadence and a freshness target. Behavioural features decay with time so that stale signals lose weight.

3.4Causal and predictive services

Four families of models sit in this layer.

• Heterogeneous treatment effects. For decisions that allocate exposure, we estimate how the effect of an action varies across cohorts. Tree-based estimators with honest sample splitting give valid confidence intervals for these effects without sparsity assumptions².
• Causal graphs and adjustment. Each decision has a written causal graph; it states which confounders must be adjusted for and which assumptions are untestable from data alone¹.
• Synthetic control. Where a change is rolled out to a whole region and no randomised holdout is possible, the design estimates the counterfactual from a weighted combination of untreated regions⁸.
• Prediction. Gradient-boosted trees for tabular risk and propensity scores⁹, Transformer sequence models for session behaviour where the latency budget allows¹⁰, and classical time-series models for demand forecasting¹¹. Every predictive model is evaluated on data it was not trained on, because a model fitted to historical data can match it closely and still generalise poorly¹².

3.5Decision engine

The decision engine combines three mechanisms. Contextual bandits choose among candidate actions using a reward that combines conversion, margin, a customer-experience term and a risk penalty. A rule engine enforces hard policy (eligibility, regulatory rules) before any learned policy is consulted. A planner handles decisions that span several surfaces at once, for example limiting the exposure of a product with high stock-out risk while adjusting its cross-sell placement.

Exposure fairness is enforced as a constraint, not checked afterwards. In the formulation we adopt, each option must receive at least a set fraction of exposures over the horizon, and the learning algorithm is judged by regret measured against the optimal policy that meets those quotas¹³. Exploration rates are set per cohort and suppressed for cohorts with little signal or high risk.

3.6Action layer and rollout

Connectors deliver decisions to product surfaces (ranking, layout, copy variants), price and promotion rules (markdowns within guardrails), fulfilment (node selection and delivery promise), and support (proactive outreach). Every delivery uses an idempotency key, retries go through a dead-letter queue that is monitored, and every policy version is an immutable snapshot that can be restored.

A new policy runs first in shadow mode, where it computes decisions but does not serve them. It then serves a small share of traffic and widens in stages, with an automatic rollback when a latency, fairness or anomaly guardrail is breached. Each region and channel has its own kill switch, so a fault in one does not spread.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Immutable raw events, recomputable views	Derived data cannot be corrected or traced after a logic bug	6
Deduplication keys and event-time watermarks	Duplicate and late events corrupt streaming aggregates	7
Causal graph per decision	Confounded comparisons mistaken for effects	1
Honest tree-based effect estimation	Average effects hide cohorts that are harmed or not helped	2
Synthetic control for region-wide changes	No randomised holdout is available	8
Always-valid sequential tests	Continuous monitoring inflates false positives	4
Pre-period covariate adjustment (CUPED)	Experiments too noisy to detect small effects	14
Minimum-exposure quotas in the bandit	Learned policies starve some options of exposure	13
Local explanations for reviewed decisions	Operators cannot see why a model scored as it did	5
Distribution drift monitor	Inputs shift away from the training distribution	15

Source: our design, with the works cited in the table.

4Signals

The pipeline consumes four groups of signals.

• Behavioural: session depth and dwell, query reformulation, cart add and remove cadence, backtracking and repeated filter changes, error screens.
• Commerce and operations: stock levels and substitutions, discount depth and markdown cadence, carrier reliability, node congestion, the gap between promised and actual delivery, payment retries and declines.
• Data quality: field-level null rates, schema mismatches, freshness lag per feature, ingestion volume anomalies, lineage completeness.
• Trust and risk: support contacts, returns, missed service levels, identity confidence and device stability. Trust signals are included because online purchase intention depends on trust in the seller as well as on the perceived usefulness and ease of use of the site¹⁶.

Consent and residency rules differ by jurisdiction, so every signal carries the flags needed to route and filter it per region.

5Experimentation and evaluation

Before any outcome claim about this pipeline could be made, each decision policy would need to pass the protocol below.

• Pre-registration. Hypothesis, target metric, guardrail metrics and minimum detectable effect are recorded in an experiment registry before launch³.
• Sequential testing. Results are monitored with always-valid p-values and confidence intervals, so that looking at a running test does not inflate the false-positive rate⁴.
• Variance reduction. Pre-experiment values of the metric are used as a covariate, which reduces variance and lets the same test detect smaller effects or run on fewer users¹⁴.
• Heterogeneity. Treatment effects are estimated per cohort, not only on average².
• Offline before online. Policies are first evaluated offline with inverse-propensity-weighted estimates on logged decisions and with simulations of edge conditions such as stock-out spikes and payment outages, then exposed online behind the staged rollout.

The metrics follow from the decisions in scope. Revenue per session is total revenue divided by sessions. Contribution margin is revenue less variable costs (fulfilment, payment and returns). Promise accuracy is one minus the absolute gap between actual and promised delivery dates divided by the promised window, so that early and late deliveries are penalised equally. Cancellation rate is cancelled orders over orders, reported by promise-accuracy decile. The fairness gap is the largest minus the smallest estimated uplift across protected cohorts. We do not propose threshold values for these metrics: thresholds should be set from measured baselines, which we do not have.

6A worked latency budget

Decisions that render inside a page have to fit a latency budget. Table 2 adds per-stage design targets to show how the end-to-end target is split. The inputs are design targets, not measurements.

Stage	Target range (ms, input)	What it covers
Ingest	10 to 20	Accept and validate the triggering event
Feature read	20 to 30	Fetch online features for the request
Decision	25 to 40	Score candidates and apply policy
Render	20 to 30	Return the decision to the surface
Sum of stage targets	75 to 120	Computed: sum of the rows above
End-to-end p95 target	100	Input

Source: Worked model; inputs in the table.

The arithmetic shows a tension in the targets themselves: if every stage runs at the top of its range, the sum is 120 ms, above the 100 ms end-to-end target. The budget only holds if at most some stages run near their upper bound at once. Adding stage targets is also only a planning device: the 95th percentile of a sum is not in general the sum of the 95th percentiles, so the end-to-end figure has to be measured directly. When it is exceeded for a sustained period, the decision service falls back to a lighter policy.

7Observability and governance

The observability plane tracks four kinds of health.

• Decision health: latency, timeout rate and fallback rate. A rising fallback rate is an early sign of instability.
• Data health: freshness, null spikes and schema mismatch counts.
• Model health: calibration, stability of uplift estimates, and drift. Drift is measured as the Kullback-Leibler divergence between the current and a reference feature distribution¹⁵.
• Fairness and compliance: fairness gaps, residency routing correctness and consent coverage.

Each request carries one identifier from ingest through feature read, decision and render. The model version, policy hash, feature snapshot hash and cohort tags are logged per call, and a replay harness can re-run any logged decision for an audit or an incident review.

Governance rests on an experiment registry with pre-committed guardrails, a data catalogue with owners and sensitivity classes, role-based access with approval for new sensitive signals, and data protection impact assessments for new pipelines. Incident playbooks follow one shape: detect, contain by falling back, diagnose, fix, restore, review. A fairness breach halts exposure for the affected policy until the gap is analysed by cohort and the constraints are adjusted.

On privacy, the design collects only fields that a decision uses, hashes personal identifiers, keeps records in their region, and offers an opt-out from personalisation. Where an offer or ranking is personalised, the surface can say why it appears.

8Illustrative scenarios

The scenarios below are hypothetical. They show how the layers would interact; they are not results.

Delivery promise shaping. If a fulfilment node became congested, the operations signals would raise the predicted gap between promised and actual delivery for products served from that node. The planner would then shorten the exposure of fast-ship badges for those products and widen the promised window. Whether this would lower cancellations is exactly what a staged, pre-registered experiment would have to show.

Ranking with exposure quotas. A ranking policy learning from clicks would tend to concentrate exposure on a few products. With minimum-exposure quotas in the bandit¹³, every eligible product would keep a floor of exposure, and the cost of that floor would appear as measurable regret against the unconstrained policy.

Friction-triggered support. A session with repeated payment retries and navigation loops would cross a friction score threshold and trigger an offer of help. The experiment would compare outreach against no outreach on later purchase and contact rates, with a guardrail on complaint rate.

9Limitations

• No measurements. We have not measured the effect of this pipeline on any outcome. Every benefit implied by the design is a hypothesis to be tested with the protocol above.
• Causal identification. Adjustment for confounding depends on a causal graph that cannot be fully checked from data. Unobserved confounders remain, and sensitivity analysis can bound their effect but not remove it¹.
• Sparse cohorts and new regions. Effect estimates for small cohorts are noisy, and cautious exploration there slows learning.
• Attribution. Cross-device journeys and incomplete identity resolution blur which action caused which outcome.
• Latency budget. As Table 2 shows, the stage targets do not sum within the end-to-end target in the worst case. The budget is unvalidated.
• Thresholds. Fairness, drift and freshness thresholds are left as parameters; setting them needs baselines we do not have.
• Operator effects. Outcomes depend on how operators use explanations and approve policies. That is a human-factors question this design does not answer.
• Model risk. Adversarial traffic and manipulated inputs could steer learned policies. The design detects drift; it does not prevent manipulation.

10Conclusion

We have described a pipeline that treats analytics as a control loop: contract-checked events, recomputable features, causal and predictive services, a constrained decision engine, and staged action with rollback, all traceable from decision back to source event. Each part rests on published methods, summarised in Table 1. What the design does not yet have is evidence of its own effect. The next step is to run the evaluation protocol on one decision surface, starting with delivery promises, and report what is measured.