When Does Real-Time Actually Matter for AI?

Real-time data can improve a decision. It can also add continuously running compute, more writes, harder recovery, and another source of training-to-serving differences. The right question is not whether a feature is "real-time." It is: How often does a delayed update change a decision, what is that change worth, and what does it cost to deliver the update sooner?

The answer differs by feature and use case. A fraud model may need to recognize a burst of related payment attempts within seconds. A search ranker may benefit when a guest's recent browsing reaches an embedding before the next search. A model using a stable historical baseline may gain little from refreshing that baseline every second. None of those conclusions follows from the feature name alone; each needs measurement.

First, define what "fresh" means

Several clocks are often bundled into one latency number:

Suppose an account has no activity for a week. A fully caught-up streaming feature will still have a latest event timestamp from a week ago; it is current state describing an inactive account. Conversely, a new event can be seconds old yet missing from a feature that the decision just read. decision_time - latest_incorporated_event_time cannot distinguish these cases.

For an event e, track both the domain event timestamp and the source commit timestamp (when the upstream log or CDC source received the record):

pipeline_delay(e)         = time_feature_includes(e) - source_commit_time(e)
event_to_visible_delay(e) = time_feature_includes(e) - event_time(e)

Recording source commit time alongside event time is critical because event timestamps can be delayed by client connectivity, clock skew, or backdated during backfills. Separating the two keeps pipeline processing latency distinct from upstream arrival latency.

For a decision, the stronger question is whether its feature values differ from what would have been available had all eligible events up to that decision arrived and been incorporated. Event-time watermarks help reason about progress, but out-of-order and late records mean that low processing delay alone does not prove completeness. Apache Flink's time model distinguishes event time, ingestion time, and processing time, illustrating the monitoring required for robust event-driven state.[1]

Measure the value of fresher state, not an assumed half-life

"Information half-life" can be a useful shorthand when a signal's contribution to a specific decision has a measured decay curve. It is not an intrinsic physical property of a feature. A change in inventory may matter immediately and remain important until the next change; a recent click may matter most for the next search; a fraud signal may become useful only when several attempts form a pattern.

If experiments show that a particular signal's incremental decision value decays approximately exponentially with delay d, one might fit:

V(d) = V(0) × exp(-λd)        half-life = ln(2) / λ

Here V(d) must be defined in terms of an outcome—such as expected avoided fraud loss or incremental bookings—and λ must be estimated for that workload. Other shapes, including thresholds, step changes, and non-monotonic effects, may fit better. An attractive curve drawn before measurement is not evidence for a freshness requirement.

The standard offline evaluation holds the model and candidate decisions fixed, reconstructs the features available at each historical decision point, and replays them with realistic added delays. Replay reveals which features, predictions, or thresholded decisions change under delay.

However, historical replay cannot usually measure business utility directly: it can show that a decision would have changed, but it cannot reveal what would have happened after an action the system didn't take. Measuring actual business impact requires an online experiment (such as an A/B test) or a valid causal evaluation design (such as off-policy evaluation). The cited study on offline versus online evaluation demonstrates why offline performance gains do not automatically translate into online business lift.[2]

Choose among ways to supply state

The choice is broader than nightly batch versus a stateful stream processor:

These approaches can be combined. Stripe describes a feature platform with both batch-only and streaming features, a lower-cost store for historical data, and a store optimized for frequent writes. Its reported p99 feature freshness of 150 ms is a measured property of that implementation, not a default promise of all streaming pipelines. Stripe also modeled batch-data delay in offline training to avoid training-serving skew.[3]

Operational cost likewise depends on event rate, cardinality, aggregate windows, data retention, existing infrastructure, cloud pricing, and the level of availability required. There is no defensible universal "streaming costs 10–25x batch" multiplier. Compare the actual incremental cost of feasible designs, including development and on-call work, over the same period.

Example: payment fraud during a burst of attempts

Consider an illustrative attack in which many payment or card-setup attempts share a device, account, or other reliably linked identifier. A count of related attempts over a short window may help a model recognize the burst before it ends. An hourly update might arrive too late. Continuous processing or a request-time read and update of a shared counter may be worthwhile.

That conclusion requires more than saying "block attempt five." The design must answer:

• What links attempts when card numbers, accounts, or IP addresses change?
• Do failed authorizations and card-setup attempts count, as well as successful payments?
• Can concurrent attempts read the same old count before any update is visible?
• How many legitimate attempts would the control block or challenge?
• What is the expected net loss from a successful fraudulent attempt, including reversals, disputes, fees, and operational costs?

Stripe describes card testing as a mix of card setups and often small payments. Its guidance recommends layered mitigations and warns that a single IP-based heuristic is insufficient.[4] Stripe Radar also emphasizes the cost of incorrectly blocking good payments.[5] Thus a useful estimate is expected avoided fraud loss minus expected harm to legitimate users and merchants—not the number of blocked attempts multiplied by an assumed payment amount.

Fast feature updates can improve fraud detection, but a duplicate payment or refund safeguard is a different problem. It needs idempotency or an atomic check and update at the decision boundary. An asynchronously refreshed stream cannot, by itself, stop two simultaneous requests from both acting on old state. Stripe documents idempotency keys for safe retries.[6]

Example: personalization after a new interaction

Personalization has no universal 15-minute freshness target. What matters is whether new activity arrives before a consequential next request. If a person searches again immediately after viewing several items, a 15-minute update schedule may miss the entire opportunity. If the next visit is tomorrow, much faster updates may have little incremental value.

Airbnb provides a measured example. It changed a guest-sequence embedding pipeline from daily batch updates to updates that often land 10–30 seconds after activity. Airbnb reported a 1.67% improvement in offline NDCG and an approximately one-third-percent increase in uncancelled bookings in online A/B tests.[7] These results support faster updates for that system. They do not tell us whether a one-minute or five-minute alternative would have achieved nearly the same outcome at lower cost; that comparison would require another experiment.

This is the right standard for other recommendation systems: measure a curve across feasible delays and connect ranking changes to user outcomes. A tiny NDCG change can be valuable at large scale, while a larger offline lift can fail to improve the business metric.

Example: AI agents and retrieved context

An agent may combine several types of state in one decision:

• Current request and conversation turns: pass them directly as request context or controlled session state.
• Recent entity activity: evaluate whether a fresh feature or event-triggered update changes the agent's answer or action.
• Documents and embeddings: refresh according to document changes and the cost of serving an obsolete version; a static corpus need not be re-embedded continuously.
• Permissions, balances, inventory, and tool locks: consult the appropriate authoritative source and enforce required consistency when taking action.

Prompt-injection detection concerns the content of a particular request or retrieved document. A tool lock concerns concurrent access. Neither becomes safe simply because a streaming feature is less than a second old. Keeping these controls distinct makes both architecture and risk review clearer.

A decision process that can be audited

1. Name the decision and outcome. Specify who or what is affected, the action taken, and the business utility of good and bad outcomes. For fraud, include false-positive costs; for ranking, include user or transaction outcomes.
2. Map required state. Identify each source, event key, update path, serving path, and consistency requirement. Separate source-event age from update delay and serving latency.
3. Measure the current system. Report pipeline delay, event-to-visible delay (recording both source commit time and event time), and serving latency as distributions. Track missing and late events, backlog, failed updates, and training-serving differences.
4. Test realistic delays. Reconstruct point-in-time features to see which predictions and decisions change under added delay. Use online experiments or valid causal evaluation designs to measure actual business outcomes.
5. Price feasible designs. Include infrastructure, storage, network, development, reliability, and on-call costs. Include the cost of controls needed for correctness.
6. Choose the design with the highest expected net value that satisfies required operational constraints. Select the architecture that maximizes expected decision utility minus total operational and infrastructure cost, subject to latency, availability, and consistency requirements (or the lowest-cost design if a fixed outcome target is explicitly specified). Revisit the choice when traffic, behavior, or system costs change.

One way to express the economic comparison is:

net_value(architecture)
  = expected_decisions_per_period × expected_utility_per_decision
    - infrastructure_cost_per_period
    - operating_cost_per_period

Choose the candidate architecture that maximizes net value while meeting required latency, availability, and consistency constraints. Expected utility should count each benefit or loss once. An organization may impose a higher return hurdle because of uncertainty or scarce engineering capacity, but no single return threshold or information half-life applies to every feature.

Takeaway

Real-time processing matters when a faster update changes a consequential decision often enough to justify its full cost. The answer has to be measured at the decision boundary. Streaming, micro-batching, batch jobs, request-time computation, and atomic state operations solve different parts of that problem; mature systems often use several of them together.

References

1. Apache Flink, Debugging Windows & Event Time and Timely Stream Processing.
2. Jeonghee Yi et al., Predictive model performance: offline and online evaluations, KDD, 2013.
3. Ben Mears, Shepherd: How Stripe adapted Chronon to scale ML feature development, Stripe, 2024.
4. Stripe, Protect yourself from card testing.
5. Ryan Drapeau, How we built it: Stripe Radar, Stripe, 2023.
6. Stripe, Idempotent requests.
7. Airbnb Engineering, The guest journey, updated in real time: extending Airbnb's sequence recommender with Chronon.