Discussion: implications of making IPP stateful via in-memory data layer

[noyitz]

The data layer work (PR #60, #64, #75) introduces in-memory state into the IPP — runtime metrics (in-flight requests, token counts) accumulated across requests and stored in-process. This is a significant architectural shift from the current stateless request/response transformation pipeline, and I think it deserves an explicit discussion before we go further.

What changes

Today the IPP is stateless — each request is processed independently, no memory between requests. With the in-memory data layer, each IPP instance accumulates runtime metrics that scorers and filters consume for model selection decisions.

Consequences of making IPP stateful

1. Replica state synchronization — With multiple IPP replicas behind Envoy, each replica accumulates its own local metrics. Without a synchronization mechanism, replicas will have inconsistent views of model state. This means we need to either build cross-replica state synchronization (which adds complexity, latency, and a consistency model to design), or accept that each replica makes decisions based on partial data. Either way, turning IPP stateful creates a new responsibility that needs to be owned — keeping distributed state consistent across replicas.

2. High availability — Stateless pods are trivially replaceable. Stateful pods are not. When a pod restarts, all accumulated metrics are lost — scorers have zero data and make blind decisions until metrics rebuild. During rolling updates, every pod goes through this cold-start period. There's no handoff mechanism between old and new pods.

3. Replica count affects correctness — With 1 replica, the data is complete. With 10 replicas, each sees ~10% of traffic. Metrics like "in-flight requests per model" become proportionally less accurate as you scale out. The more you scale (which is when you need intelligent routing most), the worse the data quality gets.

4. No consistency guarantees — Two concurrent requests hitting different replicas can make contradictory routing decisions based on their local view. Replica 1 thinks model A is underloaded and sends traffic there. Replica 2 thinks the same thing. Both send to model A, which is actually overloaded — neither knew about the other's decision.

5. Operational burden — Stateful services need monitoring for state drift, cold-start behavior, per-replica metric accuracy. Operators need to understand that IPP routing quality degrades with replica count. This is a new operational concern that didn't exist before.

6. Scope creep — Today it's in-flight requests. Next it will be latency histograms, token usage, cost accumulators, rate limit state. The IPP gradually becomes a metrics aggregation system, which is not its original purpose.

Alternatives to consider

• External shared store (Redis, memcached) — All replicas read/write to a shared store. Consistent view across replicas. Survives restarts. Adds a dependency but solves the fundamental partial-visibility problem.
• Prometheus-based — Scorers query Prometheus at selection time for aggregated metrics across all replicas. Leverages existing infrastructure already present in most K8s deployments.
• Sidecar pattern — A dedicated metrics sidecar collects and aggregates, IPP queries it locally. IPP stays stateless.
• Hybrid — Some lightweight per-instance state (e.g., rate limit headers from the last response for a specific model) combined with external aggregation for cross-replica metrics like total in-flight requests.
• Accept single-replica constraint — If IPP is always deployed as a single replica (like the upstream EPP), in-memory state works. But this should be an explicit documented constraint, not an implicit assumption.

Questions

• Has the decision to make the IPP stateful been explicitly discussed and agreed on?
• What is the expected replica count for IPP in production? Does the data layer design assume single replica?
• How do we handle multi-replica scenarios where each instance has partial visibility?
• Should the model selector's data needs drive the IPP's architecture, or should metrics collection be an external concern?

Not blocking the current work — just want to make sure this architectural direction has been intentionally chosen rather than emerging incrementally through implementation PRs.

[nirrozenbaum]

thanks for the analysis @noyitz.
these are all valid points.
we've already seen HA issues with EPP today and this in-memory state is exactly the reason why EPP cannot be scaled (the maximum we can do there is active-passive which is not effective and is a blocker for production envs).

we should be careful not to blindly repeat mistakes we did in EPP, it's definitely worth a discussion.

cc: @noalimoy the analysis above is very related to #79. would be good to have you in the loop.

[Mohammad-nassar10]

@noyitz Thanks for raising this, these are the right questions to ask before going further. Some points to clarify:

Single-replica first
We agreed to start with solving the problem for a single replica while continue thinking about HA mode (more replicas) #42.

The design is not locked to in-memory.
DataStore is an interface and Extractor is a pluggable extension point. An extractor that pulls from Prometheus, Redis, or directly from EPP pod /metrics endpoints can add or replace the in-memory one without touching the framework. The planned EPP metrics extractor (fetching live queue depth and KV cache from pods ...) is exactly an example of this. The in-memory store is the default, not the constraint.

Cold start is a real gap, in-memory counters reset on restart. But I think it is acceptable under the single-replica and for first versions requirements. Will open a follow-up issue.

The datalayer is intentionally narrow — background metrics collection for the model selector. We'll add a doc note defining that boundary explicitly.

Happy to discuss the multi-replica issue in a design doc before the next PRs go in. For more details on the datalayer see this proposal.

[ronenkat]

@noyitz Good points (and the proposal reflects for a future solution).
lets make the distinction that the stateful design we need is transient (short term), and not persistent.
IPP needs to use cross request information, e.g., model latency, prefix, ... so a solution is needed.
as @Mohammad-nassar10 pointed the mechanism in place are all about generating the data, not storing/persisting it.
In the design the only component that keep state is the Datastore, which is an interface, and can be easily revised to adopt one of the alternatives.
The key design consideration is what data the IPP needs, which is evolving (and will grow beyond the current model data). Hence the design to begin with a simple implementation in (PRs #60, #64, #75) and evolve.

[noalimoy]

One approach worth considering is Prometheus-based — I've added it as a recommended option in the (HA metrics aggregation proposal).
The idea in short: each IPP replica exposes its own local metrics as Prometheus gauges, and a small background goroutine in every replica queries Prometheus for the aggregated sum, writing it into the DataStore so Scorers always see the full picture.

What makes this work well for the concerns you raised:

• No replica sync — each replica writes only its own gauge, Prometheus sums them
  externally. Zero replica-to-replica communication.
• All replicas active — no leader, no standby, no failover. A crashed pod is simply
  excluded from the sum via Prometheus stale markers within ~5-10s.
• Scaling just works — new replicas expose their gauge, Prometheus discovers them
  automatically. Every Scorer sees the same aggregated data regardless of replica count.
• Minimal soft state — the local gauge resets to zero on restart (which is correct
  for a pod with no in-flight requests), and the aggregated cache is rebuilt from
  Prometheus within ~5s. Neither affects the global aggregated view permanently.
• No new dependencies — Prometheus is already there in most K8s environments.
• Bounded scope — IPP exposes gauges (a few lines per metric). Aggregation stays in
  Prometheus. Adding a future metric is one gauge + one PromQL line.
• Builds on proven patterns — per-instance Prometheus gauges are standard practice in projects. The aggregation
  step is native Prometheus functionality.

The main trade-off is freshness — scrape interval of ~5s means Scorers work with slightly
stale data. For relative comparisons ("model-a is more loaded than model-b") this is usually
fine. If a specific metric needs sub-second accuracy later, Redis can handle just that one
behind the same DataStore interface.