Detecting and responding to distribution shift in a deployed personalization system

1Context and motivation

A ranking or recommendation model learns from logged interactions and is then asked to score traffic that it has not seen. Predictive models generally assume that the data they are applied to come from the same joint distribution as the data they were trained on; when that assumption fails, the problem is known as dataset shift¹. In a commerce setting the assumption fails routinely. A promotion changes who arrives and what they click. A catalogue import changes which items exist and how their attributes are filled in. A new market changes the language mix of queries and product text.

The difficulty is less that models degrade than that they degrade quietly. Machine learning systems that depend on properties of their inputs tend to fail silently rather than raise an error, which is why methods for detecting shift deserve to be treated as part of the system rather than an analysis run after a complaint². Aggregate engagement metrics are a weak alarm: a sharp fall in one locale or device class can be offset by steady performance elsewhere, and outcome labels such as purchases arrive hours or days after the prediction that influenced them.

This paper describes the design we propose for detecting distribution shift in a personalization system and for responding to it. It covers which kinds of shift the system distinguishes, which detector watches each, how detection is scoped to cohorts, how fairness and latency are monitored alongside, and how each alarm maps to a response. We describe the design and the reasons for it; we have not measured its effect.

2Kinds of shift

Writing x for the inputs to a model (user, context and item features) and y for the outcome it predicts (a click or a purchase), the design distinguishes three cases, following the standard taxonomy^{1, 3}.

• Covariate shift. The distribution of inputs p(x) changes while the relation p(y | x) holds, as when a promotion brings in a new population of visitors.
• Label shift. The distribution of outcomes p(y) changes while p(x | y) holds. A seasonal rise in the base rate of purchase is an example. Label shift can be detected and its size estimated from a model's predictions on unlabelled data, without waiting for outcomes, provided the conditional p(x | y) is stable⁴.
• Concept drift. The relation p(y | x) itself changes: the same user in the same context now responds differently to the same item. It can only be confirmed once outcomes arrive^{3, 5}.

The distinction matters because the cases call for different responses. Covariate shift may need only reweighting or more exploration in the new region of input space. Label shift may need recalibration. Concept drift needs retraining on recent data, and until that is done, a fallback.

3Detectors

Each kind of shift is watched by a detector suited to the signal available when it occurs.

3.1Input and prediction distributions

Inputs and model outputs are available immediately, so they give the earliest signal. For each monitored feature and for the distribution of model scores, the monitor compares a current window against a reference window taken from the training period or from a recent stable period. For one-dimensional distributions binned into histograms, the comparison statistic is the Kullback-Leibler divergence of the current distribution from the reference⁶. Two properties of this measure shape the design: it is not symmetric, so the direction of comparison is fixed and recorded; and it is undefined when the current window has mass in a bin where the reference has none, so reference histograms are smoothed with a small pseudo-count.

Divergence on single features misses changes in the joint distribution, and embedding features are high-dimensional. For these the design uses a two-sample test. The kernel two-sample test compares two samples through the distance between their mean embeddings in a reproducing kernel Hilbert space and derives tests with a stated significance level from it⁷. An empirical comparison of shift detectors found that two-sample testing applied after dimensionality reduction with a pre-trained classifier outperformed the other detectors it compared across the shifts studied, which is the configuration we adopt for embedding inputs: test on a low-dimensional projection, not on raw vectors².

3.2Label distribution

Before outcomes arrive, the monitor estimates the current label distribution from the confusion matrix of the deployed model on held-out data and its predictions on current traffic, which is the black box shift estimation approach⁴. A significant change in the estimated base rate raises a label-shift alarm and triggers recalibration review rather than retraining.

3.3Error rate once outcomes arrive

When outcomes are joined to predictions, the monitor tracks the model's online error. Two families of detectors fit this stream. The drift detection method treats the error rate as a binomial quantity that should stay level or fall while the distribution is stationary, and defines a warning level and a drift level from its running mean and standard deviation⁸. ADWIN keeps a window of recent values whose length is recomputed from the observed rate of change, removing the need to choose a fixed time scale, and comes with bounds on its false positive and false negative rates⁹. For metrics that are not error rates, such as click-through on a surface, the monitor uses a cumulative sum chart, which accumulates deviations from a target level and signals when the sum crosses a threshold¹⁰.

4Cohort-level monitoring

Every detector above runs per cohort, not only on the whole population. Cohorts are defined by locale, language, device class, traffic source and recency of the user's last visit. The reason is the one given in the introduction: aggregate statistics hide local failures, and a shift confined to one language or one device class is exactly the kind a global average absorbs.

Cohort monitoring has a cost. Each cohort and metric pair is a separate test, run at every evaluation interval, and the number of false alarms grows with the number of tests. Table 3 works through this. The design responds in three ways: it uses sequential detectors with explicit false-alarm control rather than repeated fixed-sample tests, it requires an alarm to persist for a set duration before it triggers an action, and it ranks concurrent alarms by the traffic share of the affected cohort so that operators see the largest first.

5Fairness and latency as shift signals

Shift does not affect all groups equally. A change in language mix, for example, may lower relevance for speakers of one language while leaving the aggregate unchanged. The monitor therefore tracks a parity metric across protected cohorts alongside the shift detectors. For outcome-based parity it uses the equal opportunity criterion, which compares true positive rates across groups, because it conditions on the outcome rather than requiring equal rates of positive prediction¹¹. A parity gap that exceeds its threshold for the persistence window halts further rollout of the affected policy until the gap is analysed.

Latency is treated as a reliability signal in the same monitor, not a separate dashboard. A shift in traffic can change latency as well as accuracy: a promotion raises load, a catalogue import enlarges candidate sets. In a large service, rare slow responses in components become common at the level of whole requests that fan out across many of them, so tail percentiles rather than means are the quantity to watch¹². The monitor follows the practice of tracking latency, traffic, errors and saturation together for each serving path¹³, and a sustained breach of the tail-latency target switches the path to its fallback.

6Response

Each alarm maps to a response that has been approved in advance, so that containment does not wait for a meeting.

• Throttle. Exposure of the affected policy is reduced for the alarming cohort only. Other cohorts are untouched.
• Fall back. The cohort is served by a warm rule-based policy (popularity within category and locale, with business rules applied) that has no learned dependency on the shifted inputs.
• Recalibrate. For label-shift alarms, model scores are recalibrated to the estimated current base rate, following the correction that accompanies the estimate⁴.
• Retrain. For confirmed concept drift, the model is retrained on a window weighted toward recent data and validated in shadow mode before it serves traffic.
• Roll back. If a new model version is the cause, the previous version is restored from its immutable snapshot.

Restoration after a fallback or rollback is staged: the repaired policy runs in shadow, then serves a small share of the cohort, and widens only while guardrail metrics hold. Guardrail metrics are those that a change must not harm, declared before the change is exposed, which is how controlled experiments limit damage from a bad variant¹⁴. Every incident is written up with its trigger, affected cohorts, time to detect, time to recover and the corrective action, in the blameless postmortem form used in site reliability practice¹³.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Separate handling of covariate, label and concept shift	One response applied to every kind of shift	1, 3
Smoothed KL divergence on feature and score histograms	Early warning before outcomes arrive	6
Kernel two-sample test on reduced embeddings	Joint and high-dimensional shifts missed by per-feature checks	7, 2
Black box label-shift estimation	Base-rate change detected only after labels arrive	4
DDM and ADWIN on online error	Concept drift confirmed with controlled false alarms	8, 9
CUSUM on engagement metrics	Small persistent changes lost in noise	10
Equal opportunity parity per protected cohort	Shift that harms one group but not the average	11
Tail-latency and saturation monitoring per path	Load shifts that degrade responses before accuracy	12, 13
Guardrail metrics on staged restoration	A repaired policy that harms another metric	14

Source: our design, with the works cited in the table.

7Alarm parameters

The detectors need thresholds and windows. We do not have baselines from which to set them, so Table 2 lists the initial values the design starts with as explicit inputs. Each is a placeholder to be replaced once a reference period of production data has been collected and the false-alarm rate of each detector has been measured against it.

Signal	Parameter (input)	Initial value (input)	Action on breach
Feature freshness	Share of critical features refreshed in the last 24 h	below 90%	Alert
Score distribution	KL divergence from reference, per cohort, hourly	above 0.08	Investigate
Fairness	Equal opportunity gap between protected cohorts	above 5 percentage points for 6 h	Halt rollout
Tail latency	p95 / p99 per serving path	above 110 ms / 160 ms	Fall back
Fallback share	Share of requests served by fallback, steady state	above 3%	Review

Source: Worked model; inputs in the table.

8A worked false-alarm model

Table 3 shows why cohort-level monitoring needs explicit false-alarm control. If every cohort and metric pair is tested independently at each interval with a fixed false-positive rate, and no shift is present, the expected number of false alarms per day is the product of cohorts, metrics, intervals per day and the per-test rate.

Row	Quantity	Value
1	Cohorts monitored (input)	40
2	Metrics per cohort (input)	5
3	Evaluation intervals per day, hourly (input)	24
4	Per-test false-positive rate (input)	0.001
5	Tests per day (computed: rows 1 x 2 x 3)	4,800
6	Expected false alarms per day (computed: row 5 x row 4)	4.8

Source: Worked model; inputs in the table.

Even at a per-test rate of one in a thousand, this configuration produces about five false alarms a day, enough to train operators to ignore the monitor. Lowering the per-test rate to one in ten thousand brings the expectation to about 0.5 a day at the cost of slower detection. The model assumes independent tests, which overlapping cohorts and correlated metrics violate, so it is an illustration of scale rather than a forecast. It motivates the three choices described above: sequential detectors with stated false-alarm bounds^{9, 10}, a persistence requirement before action, and ranking by affected traffic.

9Illustrative scenarios

The scenarios below are hypothetical. They show how the parts would interact; they are not results.

A promotion changes the visitor mix. A sitewide sale would bring in visitors whose features differ from the training population. The score-distribution monitor would raise a covariate-shift alarm within hours, before outcome labels exist. If the error-rate detectors stayed quiet once purchases arrived, the shift would be logged as covariate only and the response limited to wider exploration for the new cohort.

A language mix change in one market. A rise in queries in a second language in one market would show first in the two-sample test on query embeddings for that locale cohort, while the global aggregate stayed flat. If the parity monitor then showed a widening equal opportunity gap for that language group, rollout of the current policy in that cohort would halt and the cohort would be served by the fallback until a model validated on the new mix was ready.

10Limitations

• No measurements. We have not measured the detection delay, false-alarm rate or effect on outcomes of this design. Every benefit it implies is a hypothesis.
• Unvalidated parameters. The thresholds in Table 2 are starting inputs. Until they are calibrated against a reference period, the monitor's false-alarm rate is unknown.
• Assumptions behind each detector. Label-shift estimation assumes p(x | y) is stable⁴; if concept drift and label shift occur together, the estimate is biased. Divergence on marginals cannot see changes in dependence between features.
• Delayed and biased labels. Outcomes in personalization are observed only for items that were shown, so the error rate the detectors watch is itself shaped by the policy. Concept drift in unexposed regions of the catalogue can go unseen.
• Multiple testing. Table 3 shows the scale of the problem under independence; real cohorts overlap, and the design does not give a closed-form false-alarm rate for the whole monitor.
• Fallback quality. A rule-based fallback is safer but less relevant. The cost of time spent on fallback is unmeasured.
• Adversarial inputs. The detectors are designed for natural shift. Deliberate manipulation of inputs could be crafted to stay under every threshold.

11Conclusion

We have described a design that treats distribution shift as an expected operating condition of a personalization system: three kinds of shift kept distinct, a detector matched to each and to the signal available, detection run per cohort with explicit false-alarm control, fairness and tail latency monitored in the same place, and a pre-approved response for every alarm. Each part rests on published methods, summarised in Table 1. What the design does not yet have is evidence of its own behaviour. The next step is to collect a reference period, calibrate the parameters in Table 2 against it, and report the measured false-alarm rate and detection delay of each detector.