Real-time analytics for high-velocity commerce: an event-time streaming design

1Context and motivation

Some commerce decisions lose most of their value if they wait for a batch job. During a flash sale, stock can run out before an hourly report shows the trend. A product that suddenly becomes popular needs its inventory, ranking and advertising adjusted while the demand is still there. Fraud attempts cluster in the same high-traffic periods, when manual review has the least capacity. Batch analytics answers these questions correctly but late.

This paper describes the architecture we propose for keeping time-critical commerce decisions on a streaming path. The decisions in scope are stock visibility and oversell protection, surge detection for demand and traffic, fraud scoring of orders and payment attempts, and the in-session ranking and checkout choices that depend on recent behaviour. We describe the design and argue each part from published work. We do not report its effects, because we have not measured them.

2Design principles

The design follows six principles.

• The log is the source of truth. Every state change is appended to a durable event log; every table, aggregate and feature is derived from it and can be rebuilt by replaying it^{1, 2}.
• Event time, not arrival time. Aggregates are computed over the time at which an event happened, and the system states explicitly when it considers a window complete³.
• One engine for stream and replay. The same code computes live aggregates and historical backfills, so the two cannot drift apart⁴.
• Latency is budgeted, not hoped for. Each stage of a decision has a latency target, and the end-to-end tail, not the average, is what the budget constrains⁵.
• Degrade, do not fail. Under overload the system sheds optional work and falls back to simpler decisions instead of timing out⁶.
• Integrity before freshness. A derived view may lag behind the log, but it must never lose or double-count an event².

3Event model and ordering

Every event carries an identifier, an event timestamp, an ingestion timestamp, its region and channel, and consent and residency flags. Events are immutable; a correction is a new event that supersedes an earlier one.

Producers append to partitioned topics in a durable log. Kafka was built for this pattern, collecting high volumes of user activity and operational events with low latency for both online and offline consumers¹. Partitioning by a business key (order, product, session) keeps all events for one key in order within one partition, which is the ordering most aggregates need.

Order across partitions and regions is harder. Physical clocks on different machines disagree, so the design does not use wall-clock time to decide which of two events in different regions happened first. Lamport showed that the "happened before" relation in a distributed system is only a partial order, and gave logical clocks that respect it⁷. Where the design needs to detect that two updates to the same entity were concurrent, for example two regional edits to one product's stock allocation, it attaches vector timestamps, which capture the partial order exactly at a cost that grows with the number of participants⁸. Concurrent updates are then resolved by an explicit rule for that entity type rather than by whichever write arrived last.

4Windowing, watermarks and late data

Commerce aggregates are windowed: orders per product per minute, conversion per region per five minutes, failed payments per card range per hour. In a distributed stream, events for a window keep arriving after the window has ended. The Dataflow model separates the questions of what is computed, where in event time, when results are emitted and how later refinements relate to earlier ones, and shows that correctness, latency and cost cannot all be maximised at once³.

Our design applies that model in three ways.

• Watermarks. Each stream carries a watermark, an estimate of the event time up to which input is believed complete. A window emits a first result when the watermark passes its end.
• Early and late firings. Surge and fraud windows also emit speculative results before the watermark, because a late answer is worth little there. Every window accepts late events up to an allowed lateness and emits a corrected result when one arrives.
• Accumulation mode. Downstream consumers receive either the refined total or the difference from the last result, declared per stream, so that a dashboard and a counter do not double-count each other.

Stream and replay share one engine and one definition of each window, so a backfill after a logic fix produces the same numbers the live path would have produced⁴.

5Stock visibility and oversell protection

Stock is the clearest case where a stale number does harm: a product shown as available after it sold out produces an order that cannot be fulfilled. The design keeps a per-location stock view derived from reservation, order, cancellation and receipt events.

Two requirements are kept apart. The reservation itself, decrementing available stock when an order is placed, needs a strong, single-writer check per product and location so that two buyers cannot take the last unit. The displayed availability on product and listing pages is a derived view that is allowed to lag the log, provided it never loses or double-counts a reservation. This separation of timeliness from integrity follows Kleppmann's treatment of derived data². When the derived view and the reservation check disagree, the reservation check wins and the page shows the corrected state.

6Surge detection

A demand surge is a change in the process generating orders and views, not a single large value. The design treats detection as a sequential change-detection problem on per-product and per-category rates.

• CUSUM. For each monitored rate, a cumulative sum of deviations from a reference level signals a change when it crosses a threshold. Page introduced this scheme for continuous inspection of a process; it accumulates small persistent shifts that a single-point threshold would miss⁹.
• Bayesian online changepoint detection. For rates where the pre-change and post-change regimes are both uncertain, the design maintains a posterior over the time since the last changepoint, updated online as each window closes¹⁰.

Surge signals feed stock alerts, ranking and capacity planning. The detection thresholds trade false alarms against detection delay; we leave them as parameters, because setting them needs measured baselines we do not have.

7Fraud scoring on the stream

Fraud attempts arrive as a stream with three properties that make the problem harder than offline classification. Customer habits and fraudster strategies change over time, genuine transactions far outnumber fraudulent ones, and only a small share of transactions are checked by investigators in time, so most labels arrive late or never. Dal Pozzolo and colleagues formalise fraud detection under exactly these conditions and argue that evaluation and learning must reflect them¹¹.

The design scores each order and payment attempt as it arrives, using features from short event-time windows (attempts per device, per card range, per address) and from longer profiles. Because the relation between features and fraud shifts, the model is treated as a learner under concept drift: its error is monitored on the labels that do arrive, and retraining is triggered by detected drift as well as on a schedule¹². Feedback from investigators is fed back with its delay recorded, so that recent and delayed labels are not mixed without weighting.

8In-session decisions and latency

Ranking, recommendation and checkout choices that respond to the current session have to fit within the time the page takes to render. Latency is not neutral for users: in a controlled study and in a large web-search log, Arapakis and colleagues found that users notice added delays beyond a threshold and are more likely to click on a result page that is served faster¹³. We do not carry their numbers across to commerce; we take from them only that response time is itself a factor in user behaviour and has to be budgeted.

A decision request fans out to several services: feature reads, candidate scoring, policy checks. When a request waits on many parallel calls, the slowest of them sets the response time, so rare slow calls in each service become common slow responses overall. Dean and Barroso describe techniques for tolerating this variability, such as hedged requests to replicas and returning a good-enough result when part of the work is late⁵. The design applies both: feature reads are hedged after a short delay, and the scorer returns a fallback ranking when its budget expires.

Table 1 summarises the components, the problem each addresses and the evidence behind it.

Component	Problem it addresses	Evidence
Append-only partitioned event log	Derived data cannot be rebuilt or traced	1, 2
Event-time windows with watermarks	Arrival order and delays distort windowed aggregates	3
One engine for stream and replay	Live and backfilled numbers diverge	4
Logical and vector clocks	Wall clocks cannot order events across regions	7, 8
Single-writer reservation, lagging display	Oversell versus stale availability	2
CUSUM and online changepoint detection	Surges detected late or confused with noise	9, 10
Drift-aware fraud scoring, delayed labels	Fraud patterns shift; labels arrive late and imbalanced	11, 12
Hedged reads and fallback results	Tail latency of fan-out requests	5
Load shedding and graceful degradation	Overload spreads into cascading failure	6

Source: our design, with the works cited in the table.

9A worked latency budget

Table 2 splits a target for one in-session decision across its stages. Every value is a design target that we chose as an input to the model; none is a measurement.

Stage	Target (ms, input)	What it covers
Request routing	10	Edge to the regional decision service
Feature read	25	Online features from the regional store, hedged
Scoring and policy	35	Score candidates and apply policy rules
Response assembly	15	Serialise and return the decision
Sum of stage targets	85	Computed: sum of the rows above
End-to-end p99 target	100	Input
Headroom	15	Computed: end-to-end target less the sum of stages

Source: Worked model; inputs in the table.

The headroom row is the margin left for queueing and network variance. Adding per-stage targets is a planning device only: the 99th percentile of a sum of latencies is not the sum of their 99th percentiles, and fan-out makes the tail worse than any single stage suggests⁵. The end-to-end figure therefore has to be measured directly, and the fallback ranking is served whenever the budget expires.

10Overload and degradation

Traffic peaks are when real-time analytics is most needed and overload most likely. The design follows the practice described for large production services: each request carries a criticality, the service rejects or simplifies low-criticality work before it exhausts capacity, and retries are bounded so that a struggling service is not pushed into cascading failure⁶. In this design the order of shedding is fixed in advance: first optional enrichments (secondary recommendations, non-critical dashboards), then personalisation (falling back to a non-personalised ranking), and never reservation checks or fraud scoring on payment.

Service level objectives are defined per decision path, on the tail latency and the share of requests that receive a full rather than a fallback decision, and are monitored per region⁶.

11Observability

The streaming path is monitored for four kinds of health:

• Freshness: the gap between the newest event time processed and the current time, per stream and region.
• Completeness: the share of events arriving after their window's watermark, and the volume of late corrections emitted.
• Decision health: tail latency, fallback rate and shed rate on each decision path.
• Model health: fraud-model error on arriving labels and drift indicators on its inputs¹².

Each decision is logged with the event offsets, window results and model version that produced it, so it can be replayed from the log for an audit.

12Illustrative scenarios

The scenarios below are hypothetical. They show how the parts would interact; they are not results.

A flash sale. When a sale opens, order rates for the featured products would rise sharply. The CUSUM statistic on those rates would cross its threshold within a few windows, raising stock alerts and moving ranking exposure away from products near zero availability. Reservation checks would still refuse a second order for the last unit while the displayed availability lagged.

Sudden popularity. A product with little history would start receiving views and add-to-cart events from a new referral source. The online changepoint posterior would shift towards a recent change, and the surge signal would reach replenishment planning. Whether this signal arrives early enough to change a stock decision is what a measured evaluation would need to show.

A card-testing burst. Many small payment attempts from a narrow card range would raise short-window counts in the fraud features. Scoring would flag them for review or decline, and the investigators' labels, arriving later, would be fed back with their delay recorded.

13Limitations

• No measurements. We have not measured the effect of this design on conversion, stock-outs, oversell, fraud losses or latency. Every benefit implied here is a hypothesis.
• Watermarks are estimates. A watermark can be wrong in both directions: too early and windows are corrected often; too late and results are delayed. The allowed lateness is a parameter with no measured basis yet³.
• Detection thresholds. CUSUM and changepoint thresholds trade false alarms against delay. They need baselines per product category that we do not have.
• Delayed and missing fraud labels. Most transactions are never verified, so the fraud model's measured error is biased towards the cases investigators chose to check¹¹.
• Latency budget. The figures in Table 2 are inputs, not observations. The end-to-end tail has not been measured.
• Cross-region ordering. Vector timestamps grow with the number of writers; the design limits them to entities with few writing regions, and conflict rules for other entities are not yet specified⁸.
• Cost. Running stream processing continuously costs more than batch for the same aggregates. Which decisions justify that cost has not been evaluated.

14Conclusion

We have described a streaming design for time-critical commerce decisions: an append-only event log, event-time windows with watermarks, logical ordering across regions, sequential change detection for surges, drift-aware fraud scoring, and a decision path with a latency budget and planned degradation. Each part rests on published methods, summarised in Table 1. The design does not yet have evidence of its own effect. The next step is to measure freshness, late-event rates and decision tail latency on one stream, starting with stock reservations, and report what is found.