Operating a personalization system across markets: launch, experimentation, rollout and incident response

1Context and motivation

Most of the literature on recommender systems is about models: how to predict what a person will want next, and how to measure whether the prediction is good¹. Much less of it is about the work that follows once a model is in production in several markets at once. That work decides whether a model that looked good offline ever helps anyone, and whether a model that goes wrong is caught before it does damage.

Running one personalization system across markets raises operational problems that a single-market deployment does not. Catalogues turn over at different speeds. Traffic mix, device mix and language mix differ. A promotion in one market changes the data the shared model learns from in all of them. Experiments in a marketplace can contaminate their own control group. And when something breaks, the people who notice may be in a different time zone from the people who can fix it.

This paper describes the operating procedure we propose for that setting. It is the companion to two other design notes: one on governance and risk controls, and one on serving topology and latency. Here we cover the operational loop only: launch, experimentation, rollout, retraining and incident response. We describe the procedure and the reasons for each part of it. We do not report its effects, because we have not measured them.

2Scope and assumptions

The system in scope ranks and recommends products on web, app and email surfaces in several markets. A market is a country or group of countries with its own catalogue, currency and consent rules. One model family serves all markets, with market-level features and, where data allows, market-level fine-tuning. Each market has its own traffic split, its own experiment registry entries and its own kill switch.

We assume the system already has consent-aware event ingestion, a feature store and an on-path policy layer that can fall back to a rule-based ranking. Those components are described in the sibling papers. This paper treats them as given and asks how the models that run on them should be operated.

3Launch: holdouts and declared metrics

No model goes to full traffic in any market without a controlled comparison against what it replaces. The basic tool is the online controlled experiment: users are randomly assigned to the current and the candidate policy, and the difference in an agreed metric is estimated with a confidence interval².

Three choices are made before launch and written into the experiment registry.

• The overall evaluation criterion. One metric, or a small weighted set, that the experiment is meant to move. For product recommendations this is typically purchases or revenue per user over a fixed window, not clicks, because a policy can raise clicks while lowering purchases.
• Guardrail metrics. Metrics the experiment must not harm, checked alongside the target: latency, error rate, unsubscribe rate on email, return rate, and the share of traffic served by the fallback. A guardrail breach stops the experiment regardless of the target metric².
• Per-market holdouts. Each market keeps its own control group, so that an average gain across markets cannot hide a loss in one of them. A small long-running holdout, kept out of all personalization changes for a quarter, measures the cumulative effect of many small launches.

Experiments also check their own integrity. A sample ratio test compares the observed split of users between arms with the designed split; a mismatch means the assignment or logging is broken and the result is discarded, not interpreted².

4Experiment design across markets

4.1Sequential monitoring

Operators look at running experiments every day. With fixed-horizon tests, repeated looks inflate the false-positive rate, so a team that stops the first time a result looks significant will ship many changes that do nothing. The procedure uses always-valid p-values and confidence intervals, which stay valid however often the results are checked³. Stopping early for harm is always allowed; stopping early for a win is allowed only under the sequential test.

4.2Variance reduction

Small markets produce small samples. Using each user's pre-experiment value of the metric as a covariate reduces the variance of the estimated effect, so the same test can detect a smaller effect or reach a decision with fewer users⁴. This matters most in the markets where traffic is lowest, which are also the markets where a model trained mostly on larger markets is most likely to fit poorly.

4.3Interference in marketplaces

User-level randomisation assumes that the treatment affects only treated users. In a marketplace this often fails: if the candidate policy steers treated users towards a scarce item, control users find less of it. Blake and Coey show this with an eBay email campaign, where ignoring test-control interference overstated the campaign's effectiveness by a factor of around two, and the bias was larger where supply was less elastic⁵. For policies that shift demand between items with limited stock, the procedure therefore randomises at a coarser level (by market, by time block, or by product cluster) and accepts the loss of statistical power that comes with it.

4.4Exploration under changing content

A ranking model trained only on what earlier policies chose to show cannot learn about items it never showed. Catalogues that change daily make this worse, because new items have no history. Contextual bandit methods address it by choosing items from user and item context while adapting to click feedback, and a bandit policy can be evaluated offline from logs of randomly served traffic before it is exposed to users⁶. The procedure keeps a small randomised exploration slice in each market for that reason: it provides the unbiased logs that offline evaluation needs.

4.5Beyond accuracy

A policy that maximises predicted relevance tends to show the same items to everyone and the same items to one person repeatedly. The recommender literature treats diversity, novelty, serendipity and catalogue coverage as objectives in their own right and finds that the relationships between them are not yet well understood⁷. The procedure reports intra-list diversity and catalogue coverage per market as secondary metrics in every experiment, and enforces a per-user frequency cap on repeated exposure of the same item. The cap values are our design parameters, not results.

5Staged rollout and rollback

A policy that passes its experiment still goes out in stages. It runs first in shadow mode, where it computes decisions but does not serve them, so that its latency and output distribution can be compared with the live policy on real traffic. It then serves a small share of traffic in one market, widens to a larger share, and only then extends to other markets. Each stage has exit criteria: no guardrail breach, no sample ratio mismatch, and stable fallback share.

Rollback is automatic when a guardrail is breached and manual otherwise. Every policy version is an immutable artefact with its configuration, so rolling back means pointing traffic at the previous version, not rebuilding it. Release engineering practice treats reproducible builds and configuration under version control as preconditions for this kind of safe, repeatable rollback⁸. Configuration needs the same care as code: machine-learning systems accumulate configuration whose errors are costly and hard to see⁹.

Table 1 maps each step of the procedure to the failure it addresses and the work it rests on.

Step	Failure it addresses	Evidence
Declared target and guardrail metrics	A launch improves one metric while harming another	2
Sample ratio test	Broken assignment or logging read as an effect	2
Always-valid sequential tests	Daily monitoring inflates false positives	3
Pre-period covariate adjustment	Small markets too noisy to detect effects	4
Cluster or market-level randomisation	Treatment and control compete for the same stock	5
Randomised exploration slice	Logged data only covers what past policies showed	6
Diversity and coverage metrics, frequency cap	Relevance-only ranking narrows what users see	7
Versioned policies and configuration	Rollback that depends on a rebuild	8, 9
Market-specific retraining cadence	The relation between features and outcomes changes	10
Blameless postmortems	The same incident recurs	8

Source: our design, with the works cited in the table.

6A worked model of exposure before rollback

The point of small early stages is to limit how many sessions a faulty policy reaches before it is caught. Table 2 makes that explicit for one hypothetical market. Every input is an assumption chosen for illustration, not a measurement.

Stage	Share of market traffic, as a fraction (input)	Market sessions per hour (input)	Time to detect and roll back, hours (input)	Exposed sessions (computed)
Canary	0.01	10,000	0.5	50
Partial	0.10	10,000	0.5	500
Half	0.50	10,000	0.5	2,500
Full, no staging	1.00	10,000	0.5	5,000

Source: Worked model; inputs in the table. Exposed sessions = share x sessions per hour x hours to detect and roll back.

The arithmetic is simple, but it shows the two levers. Exposure scales linearly with the traffic share and with the time to detect and roll back. Halving detection time has the same effect as halving the stage's traffic share. A stage that is small but slow to detect can expose more sessions than a larger stage with a fast automatic guardrail. The model also leaves out what matters most in practice: whether the fault is detectable by a guardrail at all. A policy that quietly lowers purchase rate without touching latency or errors will only be caught by the experiment, which needs days, not minutes.

7Retraining cadence

The relation between what a model sees and what users do changes over time: concept drift¹⁰. In commerce the causes are familiar, such as seasonal demand, promotions, new catalogue lines and changes in the interface itself. Models that learn from their own served traffic add a feedback loop: the model shapes the data it is next trained on, which is one of the hidden sources of maintenance cost in machine-learning systems⁹.

The procedure sets a retraining cadence per market rather than one global schedule. As initial design parameters we choose weekly retraining for markets with fast catalogue turnover, every two weeks for stable markets, and every three to four days during peak trading periods. These values are starting points, not findings. They should be replaced by cadences derived from the drift actually measured in each market, which the sibling paper on drift monitoring describes how to detect. Every retrained model goes through the same shadow, experiment and staged rollout as a new model; a retrain is a release.

8On-call and incident response

Personalization incidents are operations incidents. They need someone on call, a clear incident commander, and a written procedure, as any production service does⁸. The procedure has six steps.

1. Detect. Alerts on guardrail metrics, fallback share, latency and data freshness page the on-call engineer for the affected market.
2. Contain. The first action is to reduce exposure: fall back to the rule-based ranking for the affected market or cohort, or roll back to the previous policy version. Diagnosis comes after containment.
3. Diagnose. Check schema changes, catalogue loads, traffic mix and upstream dependencies, in that order, because data and configuration faults are more common than model faults.
4. Fix. Patch the feature or configuration, or retrain, and validate the fix in shadow mode.
5. Restore. Re-enter the staged rollout at the canary stage, not at full traffic.
6. Review. Write a postmortem.

Postmortems are blameless: they describe what happened, what the impact was, what was done and what will prevent recurrence, and they do not assign fault to individuals, because people who expect blame hide information⁸. Each postmortem records time to detect and time to recover, the markets and cohorts affected, and the follow-up actions with owners. A monthly drill exercises the fallback and rollback paths in one market so that they are known to work before they are needed.

Organisations do not reach this practice at once. A case study of a large software company describes the move from ad hoc data analysis to continuous experimentation at scale as passing through technical, organisational and business stages¹¹. We expect the same of this procedure: the registry and guardrails come first, and the postmortem discipline and per-market cadences follow.

9Limitations

• No measurements. We have not measured the effect of this procedure on any outcome, whether conversion, incident frequency or recovery time. Every benefit implied here is a hypothesis.
• Small markets. Variance reduction helps, but some markets will still be too small for user-level experiments to reach a decision in reasonable time. For those, the procedure falls back to longer tests or pooled estimates, both of which weaken the per-market comparison.
• Interference is hard to remove. Cluster or market-level randomisation reduces test-control interference but does not eliminate it, and it costs power. The size of the bias in any given market is unknown until measured.
• Guardrails catch only what they measure. Faults that do not move latency, errors or fallback share are caught only by the experiment, slowly. Table 2 assumes detection; it does not show it.
• Parameters. The rollout shares, frequency caps and retraining cadences are our design choices. They need baselines we do not have.
• People. On-call quality, postmortem honesty and the discipline to stop a winning-looking experiment early are human factors that this design depends on and does not address.

10Conclusion

We have described an operating procedure for a personalization system that runs in several markets: launch against per-market holdouts with declared guardrails, monitor experiments with sequential tests, randomise at the right level when markets share stock, roll out in stages with versioned rollback, retrain on a market-specific cadence, and respond to incidents with containment first and blameless review after. Each step rests on published work, summarised in Table 1. The procedure has not been evaluated. The next step is to run it in one market, record time to detect, time to recover and experiment outcomes, and report what is measured.