Distributed inference for personalization across regions: a serving design

1Context and motivation

A personalized ranking call usually sits on the critical path of a page: a product listing, a search result or a recommendation shelf cannot render until it returns. On a platform that serves shoppers in several regions, three forces shape where and how that call runs.

The first is distance. A model hosted in one region adds a round trip across the network for every shopper outside it. The second is data residency. The features that make a ranking personal (purchase history, recent sessions, inferred preferences) are personal data, and in the European Union transfers of personal data to third countries are permitted only under the conditions of Chapter V of the General Data Protection Regulation, such as an adequacy decision or appropriate safeguards¹. The third is variability. When one request fans out to many servers, a rare slow response on any of them becomes a common slow response for the request as a whole².

This paper describes the serving layer we propose for personalization under these forces: how requests are routed, where models and features live, how retrieval and ranking are split, how the latency tail is handled, and what the system does when a model, a shard or a whole region degrades. Governance of models and the operations of launching and testing them are outside its scope. We describe the design and the reasons for each part; we have not measured its effects.

2Design principles

• Serve in the shopper's residency region. Personal features are read and scored in the region where they are stored. Only non-personal artefacts move between regions.
• Two stages, not one. A cheap retrieval stage narrows the catalogue to a few hundred candidates and a heavier ranking stage orders them³.
• Design for the tail, not the mean. Latency objectives are stated at the 99th percentile, and the request path includes techniques that tolerate slow components rather than assuming they are rare².
• Degrade, do not fail. Every model call has a cheaper fallback, and the last fallback needs no personal data at all.
• Separate the model lifecycle from serving. New model versions load and roll back without restarting the serving process⁴.
• State budgets explicitly. Each stage of the request has a latency budget, and the sum is checked against the end-to-end objective.

3Architecture

3.1Regional topology

The unit of deployment is a serving cell. Each residency region holds a complete cell: a request router, a replica of the feature store holding that region's shoppers, a candidate index, the ranking models and the fallback tiers. The global router sends a request to the cell of the shopper's residency region, which is not always the nearest one. The cost of that choice is extra network latency for shoppers who are travelling or whose nearest point of presence is in another region; Table 3 makes the cost explicit.

Keeping personal features in-region is our design choice: we prefer not to depend on a transfer mechanism on the hot path of every request. Non-personal artefacts (model weights, catalogue item embeddings, locale-level popularity lists) are replicated to every cell, because they carry no personal data.

3.2Edge tier

Edge computing places compute and storage close to users, which can make cloud services more responsive and can mask transient outages of the central cloud⁵. It also brings its own concerns about bandwidth, management and the safety of data held at the edge⁶. In this design the edge holds only non-personal content: per-locale popularity lists and the static fallback described below. Personalized scoring stays in the regional cell, so a fault at the edge exposes no personal features.

3.3Serving layer

Between the application and the models sits a prediction serving system. Published serving systems interpose a modular layer that applies caching, batching and model selection so that applications do not depend on a particular framework⁷. Our serving layer follows the same split:

• Model versions. Each model is loaded as a versioned servable. A new version is loaded alongside the current one, traffic moves to it in steps, and the previous version stays loaded until the new one is accepted, as in TensorFlow-Serving's version management⁴.
• Adaptive batching. Requests to the ranking model are batched to use the accelerator efficiently, with the batch size bounded by the latency objective so that waiting to fill a batch never consumes the stage budget⁷.
• Prediction cache. Scores are cached per shopper context and model version for a short time. A model version change invalidates the cache, so that a cached score is never served under a different model than the one that produced it.

3.4Retrieval and ranking

Candidate generation and ranking are separate models, the split used in large production recommenders³. Retrieval is approximate nearest neighbour search over item embeddings. For catalogues held in memory on CPU, a graph index with hierarchical navigable small world structure gives logarithmic-scaling search⁸; for very large catalogues, product-quantised indexes on GPUs are an alternative⁹. The catalogue embeddings are global, but each cell filters candidates by regional availability, price list and legal eligibility before ranking.

The ranking model reads the shopper's personal features from the regional feature store and scores the candidates. A ranking model trained offline is judged first on held-out data and then online, because offline accuracy and online outcomes for recommenders can disagree¹⁰.

3.5Tail-tolerant request handling

Feature reads fan out across shards of the feature store. Dean and Barroso show that variability at the component level is amplified by fan-out, and they describe techniques that keep the overall tail short without eliminating every slow component: hedged requests, which send a second copy of a request to another replica after a short delay; tied requests, which let the replicas cancel each other; and selective replication of hot data². The design uses hedged reads for feature shards, sent once the first read has been outstanding longer than the shard's expected 95th percentile, with the number of hedges capped at a small share of traffic so that hedging does not itself overload the shards.

Retries are bounded by a per-request retry budget and every call carries a deadline that is passed downstream. Unbounded retries are a known path to cascading failure under overload¹¹.

3.6Fallback ladder

When a stage misses its deadline or a cell reports itself unhealthy, the request steps down a fallback ladder rather than failing:

1. Full model. Two-stage retrieval and ranking with fresh personal features.
2. Compact model. A smaller ranking model trained by distillation from the full model, which transfers much of a large model's behaviour into one that is cheaper to run¹².
3. Last good result. The shopper's most recent successful ranking from the prediction cache, if it is within its time limit.
4. Non-personal list. The locale popularity list served from the edge, which needs no personal data and no regional cell.

Load shedding and graceful degradation of this kind are the standard response to overload in reliability engineering¹¹. Every response records which rung served it, so that the share of traffic on each rung is visible and offline analysis can separate the effect of the model from the effect of falling back.

3.7Regional failover

If a whole cell is unavailable, its shoppers receive the non-personal list from the edge. Failing personal traffic over to another region's cell happens only where the transfer is permitted for that pair of regions; elsewhere the shopper sees a non-personalized page until the cell recovers. This trades relevance for compliance during an outage.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Serving cell per residency region	Personal features crossing borders on every request	1
Non-personal content at the edge	Distance to the cloud and transient cloud outages	5, 6
Serving layer with caching and batching	Framework coupling and accelerator underuse	7
Versioned servables	Model updates that need restarts and cannot be rolled back	4
Separate retrieval and ranking stages	Ranking the whole catalogue is too slow for the request path	3
Approximate nearest neighbour index	Exact similarity search over a large catalogue is too slow	8, 9
Hedged feature reads	Fan-out turns rare slow shards into common slow requests	2
Retry budgets and deadlines	Retry storms and cascading failure under overload	11
Distilled compact model as first fallback	The full model is too costly to run when a cell is stressed	12

Source: our design, with the works cited in the table.

4Worked models

4.1Fan-out and the tail

Table 2 shows why the design treats the tail as the main problem. If each shard independently responds slowly with probability p, a request that must wait for all N shards is slow with probability 1 minus (1 minus p) to the power N. The argument is the one made by Dean and Barroso²; the values below are our arithmetic for one illustrative input.

Shards per request, N (input)	Probability the request is slow (computed)
1	0.010
5	0.049
10	0.096
20	0.182
50	0.395
100	0.634

Source: Worked model; inputs in the table.

At 100 shards, a component that is slow one time in a hundred makes nearly two requests in three slow. Two levers follow: co-locate a shopper's features on few shards, and hedge reads so that one slow shard does not set the response time. Independence is an assumption; correlated slowness (a shared host, a garbage collection pause across a pool) makes the tail worse than this model, and hedging helps less against it.

4.2Regional latency budget

Table 3 sets per-stage design targets for two routes: a shopper served by the cell in the region where they are, and a shopper whose residency cell is in another region. The stage values and the network round trips are planning inputs, not measurements.

Stage	In-region route (ms, input)	Cross-region route (ms, input)
Network round trip to serving cell	10 to 20	80 to 150
Candidate retrieval	10 to 15	10 to 15
Feature read (hedged)	10 to 20	10 to 20
Ranking	20 to 30	20 to 30
Eligibility filters and response	5 to 10	5 to 10
Sum of stage targets (computed)	55 to 95	125 to 225
End-to-end target at p99 (input)	120	120

Source: Worked model; inputs in the table.

The in-region route fits the target even at the top of every range. The cross-region route does not fit at all: the network round trip alone can take most of the budget. For those shoppers the design either serves the compact model, which shortens the ranking stage, or serves the last good result, and it accepts that their pages are less fresh. Summing stage targets is a planning device only: the 99th percentile of a sum is not the sum of the 99th percentiles, so the end-to-end figure has to be measured directly.

5Monitoring the serving layer

Each cell reports, per model version, the 50th, 95th and 99th percentile latency of every stage, the deadline-miss rate, the share of traffic on each rung of the fallback ladder, the hedge rate and the prediction cache hit rate. The latency objective defines an error budget: while the budget remains, model changes may roll out; when it is spent, changes stop until reliability recovers¹¹. These measurements feed the measure and manage functions of an AI risk management process¹³; how that process is governed is beyond this paper.

A rising share of traffic on the lower rungs is an incident signal even when latency looks healthy.

6Evaluation protocol

Before any claim about this design could be made, a deployment would need the following.

• Load tests per cell. Measured stage latencies at expected and peak load, to replace the inputs in Table 3 with measured distributions.
• Fan-out and hedging. Measured shard latency distributions, with and without hedged reads, and the extra load hedging adds.
• Latency and outcomes. The effect of added latency on shopper behaviour in this setting, estimated with controlled slowdown experiments rather than assumed from other sites¹⁴.
• Fallback quality. Offline replay comparing the compact model and the last good result with the full model, followed by controlled experiments on the share of traffic that falls back, since offline and online results for recommenders can disagree¹⁰.
• Failover drills. Planned removal of a cell, to confirm that its shoppers receive the non-personal list and that no personal features leave their region.

7Limitations

• No measurements. We have not measured latency, availability or any shopper outcome for this design. Every benefit it implies is a hypothesis for the protocol above.
• Worked models are simplified. Table 2 assumes independent shards; real slowness is often correlated. Table 3 uses planning inputs for network round trips and stage times that vary by provider, route and load.
• Residency is simplified. We treat each shopper as having one residency region. Shoppers who move, share devices or shop across storefronts complicate routing, and the legal conditions for transfers differ by jurisdiction and change over time.
• Fallback quality is unknown. The relevance cost of each rung of the ladder is unmeasured. A ladder that falls back too often could serve worse pages than a slower full model.
• Hedging costs load. Hedged reads add traffic. Under a shard overload, hedging can make matters worse unless the cap holds.
• Cost. A full cell per region duplicates indexes and models, at a cost we have not estimated.
• Edge exposure. The edge holds no personal data in this design, but the integrity of the content it serves still has to be protected.

8Conclusion

We have described a serving design for personalization across regions: a complete serving cell per residency region, non-personal content at the edge, a serving layer with versioned models, caching and bounded batching, separate retrieval and ranking stages, hedged feature reads against the latency tail, and a fallback ladder that ends in a page needing no personal data. Each part rests on published work, summarised in Table 1, and the worked models show where the latency budget is tight. The design lacks evidence of its own behaviour; the next step is to load-test one cell, run the failover drill and report what is measured.