Multilingual embeddings for cross-region recommendations: a design for one item space across languages

1Context and motivation

A merchant who sells in several regions usually holds one catalog described in several languages. The same jacket has an English title in one storefront, a German title in another and a Thai title in a third, often written by different people and not always as direct translations. A recommender trained per market sees each of these as a separate item with its own sparse history. Behaviour learned in a mature market does not reach a new one, and a new market starts with almost no interaction data at all.

Recommender systems combine collaborative signals, learned from what users do, with content signals, learned from what items are¹. When interaction data is scarce, content is what is left, and recommending items that no user has yet interacted with is the cold-start problem². In a multi-language catalog, content only helps across markets if the content representation is itself language-independent: a product and its translation should be near each other, and two related products should be near each other regardless of the language each is described in.

This paper describes the architecture we propose for that: one multilingual embedding space for item content, shared across regions, with user representations, retrieval and ranking built on top of it. We describe the design and the reasons for each part. We do not report its effects, because we have not measured them.

2Design principles

The design follows five principles.

• One item space, many languages. Item text in every supported language is encoded into the same vector space. Market-specific behaviour is learned on top of that space, not by keeping a separate space per market.
• Content carries the cold start. In a market with little interaction history, recommendations lean on content similarity in the shared space and shift toward collaborative signals as history accumulates².
• Every language is validated before it is served. Multilingual encoders do not perform equally across languages, so a language is switched on only after it passes a per-language check.
• Retrieval and ranking are separate stages. A fast candidate stage narrows a large catalog to a few hundred items; a slower ranking stage orders them³.
• Changes reach users through experiments. A new encoder, a new language or a new ranking model is exposed through a controlled experiment with guardrails, not switched on for everyone at once⁴.

3Architecture

The design has five parts: ingest, the multilingual item encoder, user representations and candidate retrieval, ranking with policy filters, and a cold-start path for new markets.

3.1Ingest, locale tags and consent

Interaction events (views, add-to-cart, purchases, dismissals) and catalog updates enter through one ingest path. Each event is checked against a schema and tagged with the language of the surface it came from, the region of the user and the user's consent state. Events that fail the schema are quarantined rather than dropped, so they can be inspected. Events without the consent needed for personalisation are kept out of the training and feature paths. Region tags route each event to storage in its region; models that are shared across regions are trained only on data that the routing rules allow to be used for that purpose.

3.2A shared multilingual item space

Each item's title, description and key attributes are encoded by a multilingual sentence encoder into a fixed-length vector. The encoder family matters because the vectors must be comparable across languages, not only within one.

Three lines of work make this feasible. Pretraining a Transformer masked language model on text in one hundred languages produces representations that transfer across languages on a range of benchmarks, with particular gains on low-resource languages over earlier cross-lingual models⁵. A single encoder with a shared subword vocabulary, trained on parallel corpora for 93 languages, yields sentence embeddings on which a classifier trained on English data alone can be applied to the other languages without modification⁶. Dual-encoder training on translation pairs on top of a pretrained multilingual model gives sentence embeddings that retrieve the translation of a sentence across 112 languages, and the pretrained model cuts the parallel data needed for good performance⁷.

For semantic similarity, the encoder has to produce sentence vectors that can be compared by cosine similarity directly. Siamese and triplet fine-tuning of BERT produces such vectors and makes similarity search over a large collection practical, where scoring every pair with a cross-encoder would not be⁸. Where a strong monolingual or domain-tuned encoder already exists, it can be extended to new languages by distillation: a student model is trained so that a translated sentence maps to the same location as the original sentence in the teacher's space⁹. We prefer this route for commerce text because it keeps the geometry of an encoder already tuned on product language and adds languages to it, rather than starting again.

An item described in several languages gets one vector per language, plus a pooled item vector (the mean of its language vectors) for places that need one vector per item. The per-language vectors let the validation protocol below check that translations of one item land together.

3.3User representations and candidate retrieval

A user is represented by a vector in the same space, computed from the items they have interacted with, weighted by event type and recency. Implicit feedback (views, clicks, purchases) does not say what a user dislikes, and its signals differ in how much confidence they deserve; we follow the treatment of implicit feedback as preference with varying confidence, rather than as explicit ratings¹⁰. Because the user vector is built from item vectors in a shared space, a user's history in one language storefront is directly usable when the same user browses another.

Candidate retrieval finds the items nearest to the user vector. Exact nearest neighbour search over a large catalog is too slow for a request path, so the design uses an approximate index. Hierarchical navigable small world graphs give approximate nearest neighbour search with logarithmic complexity scaling and strong performance at high recall¹¹. For very large collections, GPU-based similarity search is an alternative for building and querying the index¹². The index is built per region from the items sold in that region, which keeps queries local and keeps a region's index free of items it cannot sell.

3.4Ranking and policy filters

Retrieval returns a few hundred candidates; a ranking model orders them. The two-stage split, a candidate generation model followed by a separate ranking model, follows a published design for large-scale recommendation³. The ranking model takes the similarity score from the shared space as one feature among others: the user's market, recent behaviour in that market, item availability and price in the user's currency, and the language of the surface.

Before results are returned, policy filters remove items that may not be shown in the user's region or to the user's age group, and frequency caps stop the same items from being shown repeatedly to the same user. When the ranking service is unavailable or exceeds its time budget, the surface falls back to a non-personalised list for the region, such as the items most often bought in that market recently.

3.5Cold start in a new market

When a merchant opens a new region, its items usually already exist in the shared space through their other-language descriptions, and new-language descriptions are encoded as they are written. Users in the new region have no history. The design starts these users on content-based recommendations from the shared space and on aggregate behaviour transferred from other regions, then increases the weight of collaborative signals from the new region as its interaction data grows. Combining content and collaborative information in one model is the established response to cold start²; the shared multilingual space is what lets the content side work across languages.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Multilingual pretrained encoder	Item text in many languages has no common representation	5, 6
Translation-pair dual-encoder training	Translations of one item must land close together	7
Sentence embeddings compared by cosine similarity	Pairwise scoring of a whole catalog is too slow	8
Distillation of an existing encoder into new languages	Adding a language without retraining from scratch	9
Per-language validation before serving	Encoder quality differs by language	13, 14
Confidence-weighted implicit feedback	Clicks and views are not explicit ratings	10
Approximate nearest neighbour index per region	Exact search over a large catalog is too slow	11, 12
Separate candidate and ranking stages	One model cannot both scan the catalog and rank finely	3
Content-led recommendations in new markets	No interaction history in a new region	2, 1
Controlled experiments with sequential tests	Rollout effects confused with noise or early peeking	4, 15

Source: our design, with the works cited in the table.

4Validating the embedding space per language

Multilingual models are not equally good in every language. On a set of tasks across 99 languages, multilingual BERT performs comparably to baselines in high-resource languages but much worse in low-resource ones¹³. A benchmark of 40 languages and 9 tasks finds a sizable gap between performance in English and performance after cross-lingual transfer, particularly on sentence retrieval¹⁴. Pretraining at larger scale narrows some of these gaps, but there is a trade-off between positive transfer and capacity dilution as languages are added⁵. A design that assumes the shared space is equally good everywhere would fail first in the markets with the least data to detect it.

The design therefore treats each language as something to be switched on, not assumed. Before a language serves recommendations, it passes four checks.

• Translation retrieval. For items whose descriptions exist in both the new language and a reference language, the encoded new-language description should retrieve its own item among the nearest neighbours of the reference description. This is the bi-text retrieval test used to evaluate multilingual sentence encoders^{7, 6}, applied to the catalog itself.
• Category coherence. Items in the same category should be nearer to each other than to items in other categories, in the new language as in the reference language.
• Neighbour agreement. For a sample of items, the nearest neighbours computed from the new-language vector and from the reference-language vector should overlap substantially.
• Human review. A native speaker reviews the top recommendations for a sample of items and flags mismatches the automatic checks cannot see, such as products that share words but not function.

We leave the pass thresholds for these checks as parameters. Setting them needs baselines from real catalogs, which we do not have. A language that fails stays on the regional non-personalised fallback, and its items can still be served in other languages where they pass.

5Experimentation and rollout

A change to the encoder, a new language, or a new ranking model reaches users through a controlled experiment. Each experiment declares its target metric (for example click-through or conversion from recommendations) and guardrail metrics (latency, error rate, recommendation diversity, complaint rate) before launch, and stops when a guardrail is breached⁴. Experiments are run with a holdout per region, because an effect averaged across regions can hide a harm in one market.

Results are monitored with always-valid p-values and confidence intervals, so that looking at a running experiment does not inflate the false-positive rate¹⁵. A new model runs first in shadow mode, computing recommendations without serving them, then serves a small share of traffic and widens in stages, with rollback when a guardrail is breached.

6A worked model of index size per region

The per-region index has to fit in memory on the serving hosts. Table 2 shows how its size follows from the number of items, the vector dimension and the storage precision. The inputs are illustrative, not measurements.

Quantity	Value	Kind
Items sold in the region	1,000,000	Input
Vector dimension	768	Input
Bytes per value (32-bit float)	4	Input
Language vectors stored per item	3	Input
Pooled item vectors only	3.07 GB	Computed: 1,000,000 x 768 x 4 bytes
Pooled plus per-language vectors	12.29 GB	Computed: 1,000,000 x 768 x 4 x (1 + 3) bytes
Pooled vectors at 8 bits per value	0.77 GB	Computed: 1,000,000 x 768 x 1 byte

Source: Worked model; inputs in the table.

The model shows why the design serves only pooled vectors from the request-path index and keeps per-language vectors in offline storage for validation. It also shows the size of the saving from reduced precision; whether 8-bit storage keeps retrieval quality acceptable is something to measure, not assume. The figures exclude the graph links an approximate index adds on top of the raw vectors, which grow with the number of neighbours kept per node¹¹.

7Limitations

• No measurements. We have not measured the effect of this design on recommendation quality, conversion or latency. Every benefit it implies is a hypothesis for the experiments described above.
• Benchmarks are not catalogs. The cited evidence comes from language benchmarks and parallel corpora. Product titles are short, full of brand names and units, and often not direct translations; how well these encoders behave on commerce text in each language is unmeasured here.
• Uneven language quality. Low-resource languages are where encoders are weakest¹³, and also where a merchant has the least data to detect a problem. The per-language checks reduce this risk but do not remove it.
• Cultural difference is not only language. Markets that share a language can still differ in taste, season and price sensitivity. A shared space transfers meaning, not preference.
• Transfer can carry bias. Behaviour transferred from a large market to a small one can push the small market toward the large market's tastes. The design does not yet include a fairness constraint on cross-market exposure.
• Thresholds and stage sizes. Validation thresholds, rollout shares and fallback triggers are parameters without measured baselines.
• Legal scope. Consent and residency handling are described as design choices. Whether a given routing satisfies a given jurisdiction's law is a legal question this paper does not answer.

8Conclusion

We have described a design in which catalog text in many languages is encoded into one shared embedding space, user histories and candidate retrieval operate in that space, and a separate ranking stage applies market-specific signals and policy. Each part rests on published work, summarised in Table 1. No language is assumed to work: each is validated on the catalog itself before it serves. What the design lacks is evidence of its own effect. The next step is to run the per-language validation on a real multi-language catalog, report what it measures, and then test the shared space against per-market models in a controlled experiment.