Feature freshness as a pipeline property: a design for time-sensitive personalization features

1Context and motivation

A personalization model scores a request with features: what the customer viewed in this session, what they bought last month, what an item costs now, whether it is in stock, and dense embeddings that place customers and items in a shared space. Each of those values was computed at some moment from events that happened earlier still. Between that moment and the request, the world moves on. A price changes, a promotion starts, an item sells out, a new item arrives with no history, or the customer's taste shifts. The model then scores the present with a description of the past.

Customer preferences, product perception and product popularity all change over time, and recommender models that ignore these temporal dynamics describe a customer who no longer exists¹. In the wider machine learning literature this is concept drift: the relation between the inputs and the target changes over time, and a model that does not adapt to it learns poorly^{2, 3}. Retraining addresses part of the problem. The other part sits in the feature pipeline, because a model retrained yesterday still reads whatever the pipeline serves today.

In many production systems, freshness is a side effect of a batch schedule: a job runs nightly, and features are as fresh as the last successful run. Nobody owns the gap. When a job fails or runs late, features age silently and the model keeps scoring. Changes in the external world and data dependencies are among the sources of hidden maintenance cost that machine learning systems accumulate⁴, and staleness is a direct case of both.

This paper describes a design in which freshness is a declared, measured and budgeted property of every feature. We describe the architecture and the reasons for each part. We do not report its effects, because we have not measured them.

2Design principles

The design follows six principles.

• Freshness is declared. Every feature has an owner, a refresh interval and a freshness target, recorded with its definition. A feature without a target is not served.
• Age is measured in event time. A feature's age is the time since the latest event it reflects, not the time since a job last ran, because events arrive late and out of order⁵.
• Bad data stops at the door. Inputs are validated against a schema before they can change a feature value⁶.
• One definition, two paths. A feature is computed from the same definition for training and for serving, and the two paths are compared^{7, 6}.
• Old evidence weighs less. Behavioural signals lose weight with age rather than counting equally forever⁸.
• Stale is a state, not a surprise. When a feature exceeds its freshness target, the decision layer knows and degrades deliberately.

3Architecture

The pipeline has five stages, with a freshness monitor that reads all of them.

3.1Ingest and validation

Events from the storefront, catalog, inventory and order systems enter through an ingest service that checks each payload against a versioned schema. Payloads that fail are quarantined with the reason recorded, and the producer is notified. Each event carries locale, region and consent tags, and events without valid consent do not reach feature computation.

Validation matters for freshness in a direct way. A malformed price feed that is accepted produces a fresh but wrong feature, which is worse than a stale but correct one. Errors in input data can cancel any benefit of a model, and a validation system that checks data fed to training and serving pipelines can detect such anomalies early, including skew between training and serving data⁶. Understanding, validating and cleaning data is a recurring data-management problem in production pipelines⁹.

3.2Event-time freshness accounting

Every feature value is stored with two timestamps: the event time of the latest event it incorporates, and the processing time at which it was computed. Feature age at request time is the request time minus the event time. Lag is the processing time minus the event time.

The distinction matters because data in a stream is unbounded and out of order: a pipeline never knows that it has seen all the events for a window, only that more may arrive, and each design trades off correctness, latency and cost⁵. The pipeline uses event-time windows and watermarks, and it treats late events as corrections that recompute the affected feature values. Recomputation has to be idempotent, so that replaying events after a fault does not count them twice¹⁰.

For each feature, the monitor tracks two measures. Age is reported per entity and summarised as a high percentile across entities. Refresh coverage is the share of entities whose value was recomputed within the refresh window. The window and the coverage threshold that raises an alert are parameters set per feature. For example, a catalog embedding might alert when fewer than 90% of active items were refreshed in the last 24 hours; that pair of values is an illustrative input, not a validated threshold, and it would tighten during promotions when the catalog changes faster.

3.3Feature store with online and offline parity

Computed features land in two stores: an offline store of point-in-time snapshots for training and backfills, and an online store keyed by entity for low-latency reads at serving time. Both are fed from one feature definition. Training reads the value a feature had at the time of each historical example, not its current value, so that the model learns from the same kind of input it will see when serving.

Producing fresh models continuously, while data changes over time, is one of the reasons for a shared platform with data analysis and validation components rather than ad hoc glue code⁷. The design adds a parity check: a sample of entities is scored through both paths on a schedule, and any difference in feature values or distributions is reported against the feature's owner. Tests of this kind, and monitoring of the data and models in production, are part of a rubric for production readiness of machine learning systems¹¹.

3.4Time decay of behavioural signals

Aggregates of behaviour (views, cart additions, purchases by category) are computed with an exponential decay, so each event contributes a weight of two raised to the power of minus its age divided by a half-life. The half-life is set per feature: short for session intent, longer for durable affinity. The approach follows time-weighted collaborative filtering, which gives recent data a larger weight than old data because a user's interest changes⁸.

Decay is a simple model of change, and it has a known cost. In recommender data, many characteristics shift at once and often subtly; plain time windows and instance decay discard too much signal, and models that represent temporal effects explicitly can do better¹. The design uses decay in the feature layer because it is cheap and interpretable, and leaves explicit temporal modelling to the ranking models that consume the features.

3.5Partial embedding refresh

Embeddings for items and customers are the most expensive features to recompute and the easiest to let age. The design refreshes them in two tiers. A full rebuild runs on a fixed schedule. Between rebuilds, a change log selects the entities whose inputs changed (new items, edited descriptions, price or stock changes, customers with new activity) and recomputes only their vectors, in an order set by traffic so that the entities most likely to be served are refreshed first. New items receive an initial vector from their content attributes until they have interaction history.

Production recommendation systems have gone further, training online so that embedding parameters update from customer feedback in near real time, with expiring embeddings and frequency filtering to bound memory; the authors of one such system report that system reliability could be traded for real-time learning¹². Our design stops short of online training: partial refresh keeps the training and serving paths separate, which keeps the parity check meaningful, at the cost of slower adaptation.

3.6Decisioning with freshness-aware fallback

The ranking service receives each feature with its age. If a feature exceeds its freshness target, the service does not silently use it. Depending on the feature's declared policy, it either drops the feature and uses a model variant trained without it, substitutes a coarser value (a category average in place of an item's own value), or switches the request to a rule-based fallback such as popularity within the customer's region. Every fallback is logged with the feature and age that caused it, and the fallback rate per feature is itself monitored. A rising fallback rate is a freshness incident even if no model metric has moved yet.

Table 1 summarises the components, the problem each addresses and the published work each rests on.

Component	Problem it addresses	Evidence
Schema validation and quarantine on ingest	Malformed inputs produce fresh but wrong features	6, 9
Event-time age and watermarks	Late and out-of-order events make job time a poor proxy for freshness	5
Idempotent recomputation of late corrections	Replays after a fault double-count events	10
One feature definition for training and serving, with a parity check	Training/serving skew	7, 6
Production tests and monitoring per feature	Silent data and model failures in production	11
Declared owner and freshness target per feature	Unowned data dependencies and external change accumulate as hidden debt	4
Exponential decay of behavioural aggregates	Old interactions count as much as recent ones	8
Explicit temporal effects left to ranking models	Plain decay discards signal when many factors shift at once	1
Partial embedding refresh between rebuilds	Expensive vectors age between full rebuilds	12
Freshness-aware fallback	Models keep scoring with features that no longer describe the present	2, 3

Source: our design, with the works cited in the table.

4A worked staleness budget

The freshness target for a feature only means something if the pipeline can meet it. In the worst case, a feature's age just before a refresh lands is its refresh interval plus its pipeline lag. Table 2 adds the two for four feature classes and compares the sum with the target. All values are design inputs, not measurements.

Feature class	Refresh interval (input)	Pipeline lag (input)	Worst-case age (computed)	Freshness target (input)	Within target
Session behaviour	streaming (0 min)	2 min	2 min	5 min	yes
Catalog price and stock	15 min	5 min	20 min	30 min	yes
Customer affinity aggregates	6 h	30 min	6.5 h	12 h	yes
Item embeddings, full rebuild only	24 h	2 h	26 h	24 h	no

Source: Worked model; inputs in the table.

The last row shows why the design has partial refresh. With a daily rebuild that takes two hours to land, item embeddings exceed a 24-hour target for part of every day. Either the target is loosened, the rebuild runs more often, or the entities that changed are refreshed between rebuilds. The budget also hides a distributional point: lag varies, and the worst-case age has to be measured from the lag distribution, not from a single planned value.

Table 3 shows the effect of the decay rule on one behavioural aggregate with a half-life of seven days.

Event age (days)	Weight (computed)
1	0.906
7	0.500
14	0.250
30	0.051

Source: Worked model; inputs in the table.

With this half-life, an interaction from a month ago carries about a twentieth of the weight of one from today. Whether seven days is right for a given signal is an empirical question; the design makes it an explicit, owned parameter rather than an accident of a window length.

5Evaluation protocol

Before any claim about the effect of this pipeline could be made, it would need the following evaluation.

• Freshness itself. Measure age and refresh coverage per feature against the declared targets over a full trading cycle, including promotions, and report the share of serving time each feature spends over target.
• Parity. Report the rate and size of training/serving differences found by the parity check, per feature.
• Value of freshness. For a small set of features, run a controlled comparison in which one arm serves the feature at its normal freshness and another serves it deliberately delayed, and compare ranking quality offline and engagement online. This isolates the value of freshness from the value of the feature.
• Fallback cost. Compare outcomes for requests served by fallback with matched requests served normally, to see what a stale-feature event costs.
• Decay parameters. Compare half-lives on held-out data for each behavioural aggregate.

We do not propose target values for these measurements: they would have to be set from measured baselines, which we do not have.

6Limitations

• No measurements. We have not measured the effect of this pipeline on any model or business outcome. Every benefit implied by the design is a hypothesis.
• Freshness is not correctness. A fresh feature built from wrong data is still wrong. Validation catches schema and range errors, not every semantic error.
• Parameters are guesses. Refresh intervals, targets, coverage thresholds and half-lives in this paper are illustrative inputs. Setting them needs per-feature measurement.
• Decay is a crude model of change. Exponential decay assumes interest fades smoothly. Seasonal and recurring interests, and shifts that affect many factors at once, are not captured by it¹.
• Partial refresh can bias embeddings. Recomputing only changed entities between rebuilds can leave vectors computed at different times inconsistent with one another until the next full rebuild.
• Fallbacks have their own cost. A popularity fallback is safe but generic, and it can concentrate exposure on items that are already popular.
• Cost. Shorter refresh intervals and finer age accounting cost compute and storage. The design states the trade-off but does not price it.

7Conclusion

We have described a feature pipeline in which freshness is declared, measured in event time, budgeted and acted on: validated ingest, one feature definition for training and serving, decay of behavioural signals, partial embedding refresh, and deliberate fallback when a feature is too old to trust. Each part rests on published work, summarised in Table 1. The worked budget in Table 2 shows that the targets themselves can be inconsistent with a simple batch schedule, which is the argument for treating freshness as a design property. What the design does not have is evidence of its effect. The next step is to measure age and coverage for the features in use today and run the controlled freshness comparison on one feature.