Testing checkout changes with modest traffic

1Why checkout tests are hard at small volume

Checkout is where a store loses the most buyers it has already won. Across 50 studies, the documented average cart abandonment rate is 70.22 percent, and among US shoppers who abandoned for a reason other than browsing, 19 percent did not trust the site with their card details, 17 percent found the checkout too long or complicated and 10 percent had a card declined¹. Each of those reasons suggests a change to test.

Controlled experiments are how a store learns whether such a change helps rather than guessing from a before and after comparison, which mixes the change with everything else that moved that week². The practice is well documented, but by teams that run hundreds of concurrent experiments on very large audiences³. A store with 400 checkout sessions a day faces a different constraint: most changes it could test would need months to detect. This paper sets out a design that respects that constraint.

2The design

2.1Unit and metric

The unit of randomization is the checkout session: a visitor who reached checkout, assigned once to a variant and kept there if they return. Randomizing page views would show one visitor both versions. The primary metric is completion, the share of checkout sessions that end in an order. Revenue per session is a secondary metric; it is noisier, so it needs more sessions to read.

Two guardrails sit beside the primary metric. Speed: the checkout page is held to the Core Web Vitals thresholds, which are a largest contentful paint within 2.5 seconds, an interaction to next paint of 200 milliseconds or less and a cumulative layout shift of 0.1 or less, each measured at the 75th percentile of page loads⁴. Payment: the decline rate per variant. A variant that completes more sessions while getting slower or declining more cards is not shipped without a second look.

2.2Sample size before the test

The number of sessions is set before the test starts. A common rule of thumb for about 80 percent power at a two-sided 5 percent significance level is n = 16σ²/δ² sessions per variant, where σ² is the variance of the metric and δ the smallest difference worth detecting^{2, 5}. For a completion rate p and a relative lift r, σ² = p(1 - p) and δ = p × r.

Baseline completion	2% lift	5% lift	10% lift
30%	93,334	14,934	3,734
45%	48,889	7,823	1,956
60%	26,667	4,267	1,067

Source: our calculation with n = 16p(1 - p)/(p × r)², rounded up^{2, 5}.

Take a store with 400 checkout sessions a day split evenly, so 200 per variant, and a completion rate of 45 percent. A change expected to lift completion by 10 percent needs about 1,956 sessions per variant, roughly 10 days. A change expected to lift it by 5 percent needs 7,823, roughly 39 days. A 2 percent lift needs eight months and is not testable at this volume.

The practical consequence is a filter on what to test. Changes aimed at a large, specific failure, such as a missing local payment method, a forced account creation step or a total that appears only on the last page, can move completion by enough to read. Cosmetic changes cannot, and a store should ship those on judgment or bundle several into one variant.

2.3Variance reduction

When the metric has a pre-period value for the same units, a store can subtract the part of the variance that the pre-period explains. This method, known as CUPED, adjusts each unit's metric by its pre-experiment value, and the variance falls by the factor 1 - ρ², where ρ is the correlation between the two⁶. At checkout the gain is limited, because many sessions are first visits with no history. It is worth applying to tests that run on returning customers, for example a change to saved payment methods.

2.4Stopping rule

Looking at the result every day and stopping when it crosses significance inflates the false positive rate well above the nominal level⁷. A store has two sound options. It can fix the sample size from Table 1, run in whole weeks so weekday and weekend buyers are both represented, and read the result once. Or it can use a sequential test designed for continuous monitoring, which gives a p-value that stays valid however often it is checked, at the cost of somewhat more sessions for the same sensitivity⁷.

2.5Trust checks

Before reading any result, check that the split is what the design said. A sample ratio mismatch, such as a 52/48 split over a few thousand sessions when 50/50 was planned, usually means a bug in assignment or logging, and a result from such a test should not be trusted⁵. A chi-square test on the arm counts is enough. Run an A/A test once when the setup is new: two identical arms should differ significantly in about 5 percent of runs and no more².

3Limitations

The sample sizes in Table 1 come from a rule of thumb that assumes independent sessions and a roughly normal sampling distribution; a store with very few sessions, or sessions that are not independent, needs an exact calculation. We assume a 50/50 split; an unequal split needs more sessions in total. The abandonment reasons we cite are self-reported by US shoppers and are a guide to what to test, not an estimate of what a change will achieve. We report no experiment of our own.

4Conclusion

At a few hundred checkout sessions a day, a store can run trustworthy tests, but only of changes large enough to read within weeks. Fix the unit, the metric, the guardrails, the sample size and the stopping rule before the test begins, check the split before reading the result, and spend the traffic on changes aimed at a known reason for abandonment.