Governance and safety controls for a multi-region personalization system

1Context and motivation

A personalization system decides which products a shopper sees first, which offer is shown and which message is sent, using personal data such as browsing and purchase history, location, language and device. Across regions, the rules on what data may be used, for what purpose and with what record differ, and so do the cohorts a model may treat unequally.

Controls that decide whether a model may be trained on a dataset, whether it may be promoted, who reviewed it, and how a decision can be traced afterwards are often added late, as checklists outside the pipeline. Work on machine learning in industry describes why that is fragile: data for machine learning is harder to discover, manage and version than for other software, and models are entangled with each other in ways that make them hard to treat as separate modules¹.

We describe a set of governance and safety controls that sit inside the pipeline. The scope is a multi-region personalization system: ranking, recommendations, offer selection and message timing. We describe the design and the reason for each control; we have not measured its effects. Companion papers in this series cover the serving architecture and the launch and rollout operations; this one covers governance, risk and accountability.

2Governance as a lifecycle

We organise the controls around the four functions of the NIST AI Risk Management Framework: govern, map, measure and manage². In that framework, govern is a cross-cutting function that informs the other three, and map, measure and manage apply to a specific system at specific stages of its lifecycle. The framework also lists the characteristics it treats as marks of trustworthy AI: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with harmful biases managed².

The framework is voluntary. We use it as a structure, not as a standard we claim to meet:

• Govern sets the roles, policies and records that every other control writes to: who owns a model, who may approve a promotion, and where the evidence is kept.
• Map establishes, for each personalization use, what decision it makes, whose data it uses, under what lawful basis, and which cohorts it could treat unequally.
• Measure runs the tests: data checks, model evaluations, fairness checks and experiments.
• Manage acts on the results: promotes, holds back, rolls back or retires a model, and records why.

3Controls

3.1Lawful basis and consent at ingest

Under the GDPR, processing of personal data is lawful only if at least one listed basis applies, consent for one or more specific purposes being one of them³. Where processing rests on consent, the controller must be able to demonstrate that the person consented, the person may withdraw consent at any time, and withdrawal must be as easy as giving it³. Where personal data are processed for direct marketing, the person has the right to object at any time, including to profiling to the extent it relates to that marketing, and once they object the data may no longer be processed for that purpose³.

These provisions shape the ingest layer. Each event carries the identifier of the person, their locale, and a reference to the consent or other lawful-basis record that covers the purpose for which the event will be used. An event without that reference is quarantined, so a broken consent integration shows up as a rising quarantine count. A withdrawal or an objection is itself an event: it propagates to the feature store and to the decision layer, and from that point the person's data is excluded from personalization for the affected purpose. The consent record is versioned, so a past decision can be checked against the consent state at that time.

Other jurisdictions have their own rules, so the lawful-basis check reads a per-region policy table maintained by a privacy owner; the table, not the code, changes when the rules do.

3.2Data protection by design and records

The GDPR requires the controller to implement technical and organisational measures, such as pseudonymisation, designed to implement data-protection principles such as data minimisation, both when deciding the means of processing and during the processing itself³. By default, only the personal data necessary for each specific purpose may be processed, which covers the amount collected, the extent of processing, the storage period and who can access it³.

Personal identifiers are pseudonymised at ingest, and the mapping back to a person is held separately with narrower access. Each feature declares the purpose it serves; a feature with no declared purpose cannot be read by a model. Each feature also declares a retention period, and expiry is enforced by the store. Data is stored and processed in the region whose rules apply to it.

The regulation also requires each controller to keep a record of processing activities, including the purposes of processing, the categories of data and recipients, transfers to third countries, and, where possible, time limits for erasure³. In our design that record is generated from the feature and model registries, so it changes when the pipeline changes. Where a type of processing is likely to result in a high risk to people's rights and freedoms, the regulation requires a data protection impact assessment before processing begins³. Whether a personalization use crosses that threshold is a legal judgement; as a design choice, every new pipeline and every new category of personal data needs an assessment on file before production.

3.3Documentation of data and models

Two documentation practices give the registries their content. A datasheet records a dataset's motivation, composition, collection process and recommended uses, so that people who did not build the dataset can judge whether it fits their task⁴. A model card records a model's intended use, the contexts it is not suited to, the evaluation procedure, and performance broken down by the groups relevant to the application⁵.

In our design every training dataset has a datasheet and every model version has a model card, and both are versioned with the artefact they describe. For personalization, the model card's groups are the cohorts the map function identified: region, language, device class, and new against returning customers, plus any protected attribute the system has a lawful basis to use for testing. A model card with an empty group breakdown blocks promotion.

3.4Fairness checks and their trade-offs

Fairness in personalization is not one property. In supervised prediction, equality of opportunity asks that people who truly belong to the positive class are predicted positive at equal rates across groups, and a learned predictor can be adjusted after training to meet that criterion⁶. For risk scores, three natural conditions (calibration within groups and two forms of balance across groups) cannot all hold at once except in highly constrained special cases⁷. In rankings, fairness can be stated in terms of how exposure is allocated among the items being ranked, and rankings can be computed that maximise utility for the user while meeting a stated exposure constraint⁸. The recommender systems literature treats fairness, novelty, diversity and the interests of several stakeholders as dimensions of evaluation alongside accuracy⁹.

Two design rules follow. First, every fairness check names the definition it tests: group parity of a predicted outcome, equality of opportunity, calibration within groups, or exposure share among items or providers. Because some of these cannot hold together⁷, a policy states which one it is held to and records the choice in its model card. Second, thresholds for the chosen measure are parameters set by the policy owner, not constants in the code. We do not propose values: a threshold should be set from a measured baseline and a stated tolerance, and we have neither.

3.5Experiment guardrails and holdouts

A change to a personalization model is a change to what people see, and its effect is unknown until it is measured. Online controlled experiments are the standard way to measure it, with guardrail metrics declared alongside the target metric so that a change which improves the target while harming something else is caught¹⁰. In our design every promotion that changes what shoppers see goes through a controlled experiment with a regional holdout, declared guardrails, and the fairness measure from the model card evaluated per cohort. A breached guardrail stops the exposure; rollout mechanics are in the companion operations paper.

3.6Internal audit and promotion records

An internal audit framework for AI systems proposes stages run throughout development, each producing documents that together form an audit report and that test decisions against the organisation's stated principles¹¹. Its motivation applies here: once a system is deployed, emergent problems can be hard or impossible to trace to their source¹¹.

Our design makes the promotion record the backbone of that audit. A promotion record holds the model version, the training data version and its datasheet, the model card, the test results, the experiment result and the approver. The test results include tests and monitoring checks of the kind collected in production-readiness rubrics for machine learning, which treat testing and monitoring together because the prediction behaviour of a model is hard to specify in advance¹². A model can only be served from a promotion record, and a rollback restores a previous record rather than editing the current one. Each decision logged in production carries the identifier of the record that produced it.

3.7Human review and explanation

The GDPR gives people the right not to be subject to a decision based solely on automated processing, including profiling, that produces legal effects or similarly significantly affects them, and where such decisions are allowed it requires at least the right to human intervention and to contest the decision³. Most ranking decisions are unlikely to reach that threshold, but some uses near it (eligibility for deferred payment terms, for example) may; the map function flags them, and they are excluded from automated personalization or routed through human review.

For lower-stakes uses, the design follows human-AI interaction guidelines that ask a system to make clear why it did what it did and to support efficient correction and dismissal¹³. A personalized slot can show why an item appears, and a shopper can turn personalization off. Inside the organisation, high-impact promotions (a new model family, a new data category, a new region) need an approver other than the model's author.

3.8Incident handling

When a monitor fires, the response is to contain by falling back to a non-personalized or previous policy, diagnose, fix, restore through the experiment path, and write the incident to the audit record.

4Controls and evidence

Table 1 summarises the controls, the problem each addresses, and the published work or regulation it rests on.

Control	Problem it addresses	Evidence
Lawful-basis reference on every event, quarantine without it	Personal data used without a basis for the purpose	3
Withdrawal and objection propagated as events	Data still used after consent is withdrawn or marketing is objected to	3
Purpose and retention declared per feature	Data kept or used beyond what a purpose needs	3
Records of processing generated from registries	Hand-written records drift from the real pipeline	3, 1
Datasheet per training dataset	Data reused for tasks it does not fit	4
Model card with per-cohort results	Uneven performance hidden by aggregate metrics	5
Named fairness definition per policy	Incompatible fairness goals pursued at once	6, 7, 8
Controlled experiment with guardrails and holdout	Changes that help one metric and harm another	10
Promotion record and staged internal audit	Problems that cannot be traced back to a decision	11, 12
Explanation and opt-out on personalized slots	Shoppers cannot see or correct why an item appears	13
Lifecycle structure (govern, map, measure, manage)	Controls added piecemeal with no owner	2

Source: our design, with the works cited in the table.

Table 2 maps the controls to the four framework functions.

Control	Govern	Map	Measure	Manage
Ownership, approvers, policy tables	yes
Use inventory and lawful-basis mapping	yes	yes
Impact assessment per new pipeline	yes	yes
Datasheets and model cards	yes	yes	yes
Fairness checks and experiments			yes	yes
Promotion records and audit	yes		yes	yes
Rollback, fallback, incident record				yes

Source: our design, mapped to the four functions of the NIST AI RMF².

5What an evaluation would need

We make no claim that these controls reduce harm, improve outcomes or satisfy any regulator. An evaluation would need to establish four things. First, coverage: the share of served decisions that can be traced to a complete promotion record, and the share of events carrying a valid lawful-basis reference. Second, latency of withdrawal: the time from a consent withdrawal or objection to the last decision that still used the person's data, measured end to end. Third, detection: whether fairness checks and guardrails detect injected faults, such as a deliberately skewed model in a test environment, and how quickly. Fourth, cost: the time that documentation, review and experiment add to a model change. None has been measured.

6Limitations

• No measurements. We have not measured the effect of these controls on any outcome, cost or risk.
• Not legal advice, not a compliance claim. The design reads the GDPR provisions it cites as engineering requirements. Whether a particular deployment meets them is a legal question for each controller, and other jurisdictions are handled only through a policy table we do not specify.
• Fairness is chosen, not solved. Because some fairness definitions are incompatible⁷, every policy gives something up. The design makes the choice explicit; it does not make it correct.
• Cohort data. Per-cohort results need cohort labels. Where the system has no lawful basis to hold a protected attribute, the corresponding check cannot run, and proxies carry their own risks.
• Documentation quality. The design checks that datasheets and model cards exist and that required fields are filled; it cannot check that they are true.
• Process overhead. Review and audit add delay, and if it is large teams may route around the controls. We have no estimate of it.
• Thresholds. Fairness, guardrail and data-check thresholds are parameters; setting them needs baselines we lack.

7Conclusion

We have described governance and safety controls for a multi-region personalization system: a lawful-basis check at ingest, purpose-bound and pseudonymised data, generated records of processing, datasheets and model cards, fairness checks that name their definition, experiments with guardrails, promotion records that carry an internal audit, and human review where decisions approach significant effect. Each control rests on cited regulation or published work (Table 1), and the set covers each function of the risk management framework (Table 2). The design has no evidence of its own effect yet. The next step is to measure coverage and withdrawal latency on one pipeline and report what is found.