A unified metrics framework for comparing commerce performance across regions

1Context and motivation

A commerce platform that serves stores in many countries is asked the same question again and again: is this store, category or market doing better or worse than that one? The usual answer compares raw key performance indicators (KPIs) such as average order value (AOV), conversion rate and revenue. Across regions those numbers are not on a common scale. They are recorded in different currencies, they include different taxes and fees, they come from checkout funnels with different numbers of steps, and they are produced on devices and networks of different speeds. A raw comparison mixes business performance with factors that have nothing to do with how well a store sells.

Work on online controlled experiments makes a related point inside a single product: choosing and defining metrics is a design problem in its own right, and metrics that are badly defined or misread lead to wrong decisions even when the underlying data are correct^{1, 2}. Comparing regions adds another layer to the same problem, because each region's metric definitions drift apart unless something holds them together.

We propose a unified metrics framework (UMF) for that purpose. It has six layers. Five of them remove or expose one source of non-comparability; the sixth governs how every metric is defined and changed. This paper describes the layers, the reasoning behind each, and the published work each rests on. We have not measured how much the framework changes any comparison in practice, and we state what such a measurement would require.

2Sources of non-comparability

Five sources make raw KPIs hard to compare across regions.

• Currencies. Values recorded in local currency must be converted before they can be compared, and the choice of rate matters. Short-run deviations of exchange rates from purchasing power parity (PPP) are large and volatile, and they decay slowly³. A change in a store's AOV measured in a reporting currency can therefore be mostly an exchange-rate movement.
• Price levels over time. Inflation raises nominal order values without any change in what customers buy. Separating the two needs a consumer price index⁴.
• Funnel structure. Checkout flows differ in the number and kind of steps: an added payment authentication step, a wallet that skips address entry, an instalment plan that adds an approval. A conversion rate computed over a two-step funnel and one computed over a six-step funnel do not measure the same thing.
• Performance. Slower responses change user behaviour. In web search, users notice added delay once it passes a threshold, and content-identical result pages served with lower latency receive more clicks^{5, 6}. Regions with slower networks or devices may show lower conversion for reasons outside the store's control.
• Measurement coverage. Browsers, consent rules and intermittent connectivity affect which events are recorded at all, so the same behaviour can produce different counts in different places.

None of these is a new observation. The purpose of the framework is to handle them in one place, with one set of definitions, so that every comparison on the platform uses the same treatment.

3The framework

Layers 1 to 5 transform or annotate raw metrics; layer 6 governs the definitions. Every transformed value is stored alongside the raw value it came from, never in place of it, so any comparison can be traced back to the recorded numbers.

3.1Layer 1: currency and price-level normalisation

Each monetary value is recorded in local currency with its timestamp. The framework derives three converted values, each answering a different question.

• Market-rate value, converted at the exchange rate on the transaction date: what the revenue was worth in the reporting currency.
• Constant-rate value, converted at one fixed rate for the whole comparison period: growth with exchange-rate movement held out.
• PPP-adjusted value, converted with the PPP conversion factors published by the International Comparison Program: spending relative to what money buys locally^{7, 8}.

Real values over time divide the nominal value by a consumer price index taken relative to a base period⁴:

AOV_real = AOV_nominal / (CPI_t / CPI_base)

Taxes and fees are handled before any conversion. Net AOV (NAOV) removes value-added or sales tax, duties, and payment or instalment surcharges from the order value, so that a region with tax-inclusive prices is not reported as having larger orders. Which items are removed is a definition held in the KPI dictionary (layer 6), not a choice made report by report.

3.2Layer 2: funnel standardisation

Each store's checkout events are mapped onto a canonical sequence of states: product view, cart, checkout start, payment, order. A store's own steps are assigned to these states, and where a store has several steps inside one canonical state they are merged. Movement between states is modelled as a Markov chain, following attribution work that represents customer paths as first- and higher-order Markov walks over a graph⁹. When each state can only be reached from the one before it, the standardised conversion rate is the product of the transition probabilities along the canonical path:

CR_norm = P(view -> cart) x P(cart -> checkout) x P(checkout -> payment) x P(payment -> order)

When users skip or revisit states, the product no longer equals the end-to-end rate, and the framework reports instead the probability that a session starting in the view state is absorbed in the order state, which follows directly from the chain's transition matrix.

The gain is diagnostic. Two stores with different numbers of steps report the same set of transition probabilities, so a comparison can say where in the canonical funnel they differ rather than only that their end-to-end rates differ.

3.3Layer 3: regional effects

A simple way to correct for regional behaviour is to multiply each metric by a hand-set regional weight. We do not adopt it: hand-set weights encode assumptions that cannot be checked and can hide real differences. Regional effects are estimated instead. When a KPI is compared across stores, the comparison model includes region as a grouping level in a multilevel model, so that estimates for regions with few stores are pulled toward the overall distribution while regions with many stores are estimated mostly from their own data¹⁰. Behavioural covariates such as payment mix, device mix and the share of returning customers enter as explanatory variables, and their estimated effects are reported next to the comparison, not folded silently into an adjusted number.

3.4Layer 4: performance-adjusted conversion

Because response latency changes behaviour^{5, 6}, conversion is reported both raw and stratified by performance band, for example by decile of page response time and by device class. The framework also defines a performance-adjusted conversion index:

PAC = CR / (1 + latency_penalty + device_penalty + payment_penalty)

The penalty terms are not constants we supply. Each has to be estimated, and the credible way to estimate the effect of latency on a business metric is a controlled slowdown experiment, in which a randomly chosen group receives deliberately added delay¹. Until such estimates exist for a region, PAC is not reported there and the stratified view is used instead.

3.5Layer 5: revenue integrity

Gross merchandise value (GMV) overstates revenue where refunds, chargebacks or fraud are common. The true revenue ratio (TRR) is defined as:

TRR = (GMV - refunds - chargebacks - fraud_adjustments) / GMV

It is reported with every revenue figure. Refunds and chargebacks settle weeks after the order, so TRR for a recent period is provisional; the dictionary records the settlement window after which a period's TRR is treated as final.

3.6Layer 6: the KPI dictionary

Every metric the framework produces has one entry in a versioned dictionary: name, formula, unit, currency treatment, inputs, owner, the regions where it is valid, and its status (active, deprecated or retired). The derived metrics of layers 1 to 5 (NAOV, standardised conversion, PAC, TRR) are entries like any other.

Three ideas from the experimentation literature shape the dictionary. First, metrics have roles: goal metrics that define success, driver metrics that move sooner, and guardrail metrics that protect against harm¹. Second, a goal metric should be checked for directionality (it moves the same way as the business outcome) and sensitivity (it moves enough to detect real changes), and both can be assessed with data¹¹. Third, metric values are misread even when they are computed correctly, so each entry carries its known interpretation pitfalls next to its formula².

Changes follow one procedure. A new version of a metric runs in parallel with the old one for a set period, so the break in a time series is visible and can be bridged. A deprecated metric stays computable for audit until it is retired.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Constant-rate conversion	Exchange-rate swings read as business change	3
PPP conversion factors	Spending compared without regard to local price levels	7, 8
CPI deflation	Inflation read as order-value growth	4
Markov funnel standardisation	Funnels of different lengths compared only end to end	9
Multilevel regional model	Hand-set regional weights; noisy estimates for small regions	10
Performance-stratified conversion	Latency differences read as store performance	5, 6
Slowdown experiments for PAC penalties	Penalty weights with no evidence behind them	1
Metric roles (goal, driver, guardrail)	One metric asked to do every job	1
Directionality and sensitivity checks	Goal metrics that do not track the outcome	11
Interpretation pitfalls in each entry	Correct numbers misread	2

Source: our design, with the works cited in the table.

4A worked model: order value in two regions

Table 2 shows how the conversions of layer 1 change a comparison of AOV between two hypothetical regions. Region B prices in the reporting currency; Region A does not. All inputs are illustrative, not observed.

Row	Region A	Region B	Type
Nominal AOV, tax-inclusive (local currency)	1,000	60	Input
Tax rate included in the price	10%	20%	Input
Market exchange rate (local currency per USD)	20	1	Input
PPP conversion factor (local currency per international $)	10	1	Input
NAOV (local currency)	909.09	50.00	Computed: AOV / (1 + tax rate)
Raw AOV at market rate (USD)	50.00	60.00	Computed: AOV / market rate
NAOV at market rate (USD)	45.45	50.00	Computed: NAOV / market rate
NAOV at PPP (international $)	90.91	50.00	Computed: NAOV / PPP factor

Source: Worked model; inputs in the table.

On raw AOV at market rates, Region B's orders look 20% larger than Region A's (60.00 against 50.00). Removing tax narrows the gap to 10% (50.00 against 45.45). In purchasing-power terms the order reverses, and Region A's orders are about 82% larger (90.91 against 50.00). Each figure is correct for a different question. Market-rate NAOV says what the orders are worth to a business that reports in US dollars; PPP-adjusted NAOV says how large the orders are relative to what money buys locally. The framework reports both and labels each; it does not pick one number for every purpose.

5Evaluation a deployment would need

Before any claim that the framework improves comparisons, it would have to be tested on a panel of stores with known histories. Three checks follow from the design. The first is decomposition: for a set of KPIs, how much of the between-region variance each layer removes, reported layer by layer so that no single adjustment hides the others. The second is a check against known differences: where a real business change is documented (a price change, a new payment method), the adjusted metric should still show it. A normalisation that removes known effects is removing signal, not noise. The third is stability: a metric's ranking of regions should not swing with exchange-rate movements once constant-rate conversion is applied.

6Limitations

• No measurements. We have not measured how much any layer changes cross-region comparisons. Every benefit implied here is a hypothesis for the evaluation above.
• PPP is an aggregate. PPP factors are produced for gross domestic product and its expenditure aggregates⁷, not for the basket a given store sells, so a PPP-adjusted value is an approximation for any single category.
• Funnel mapping is a judgement. Assigning a store's steps to canonical states is done by people or rules that can be wrong, and a wrong mapping moves probability between states.
• Latency evidence comes from search. The cited latency studies concern web search^{5, 6}. That checkout behaviour responds the same way is an assumption, which is why PAC waits for slowdown experiments.
• Partial pooling shrinks. A multilevel model pulls small regions toward the overall mean¹⁰. A small region that really is different can look more ordinary than it is.
• Provisional revenue. TRR for recent periods changes as refunds and chargebacks settle.
• Coverage is recorded, not corrected. No layer corrects for events that were never recorded; the framework can only report coverage alongside each metric.
• Normalisation answers a chosen question. Every adjusted figure answers one question and not others. Showing raw and adjusted values together reduces the risk of misreading but does not remove it².

7Conclusion

Comparing commerce performance across regions requires agreeing on what is being compared. We have described a framework that converts currencies and price levels in labelled ways, standardises funnels as Markov chains, estimates regional effects rather than assuming them, separates performance from store behaviour, accounts for revenue that does not stick, and holds all of it in one versioned dictionary. Each part rests on published methods, summarised in Table 1. The framework has not been evaluated; the next step is to run the decomposition and known-difference checks on a panel of stores and report what they show.