What
Add a datalayer extractor that observes request/response events and maintains, per model, a rolling view of time-to-first-token (TTFT) under load: a load-invariant service floor (P10Low), the current median (P50), and the average in-flight requests at that median (inflightAtP50).
Why
Routing decisions need to know how a model behaves under its current queue, not just a static latency number. These three values let a scorer predict the TTFT of the next request at the current in-flight level.
How
Each response's measured TTFT is paired with the queue depth it saw at dispatch (correlated request -> response via CycleState) and stored in a fixed-size sliding window. Percentiles are recomputed periodically and published to the model attribute store for consumers.
What
Add a
datalayerextractor that observes request/response events and maintains, per model, a rolling view of time-to-first-token (TTFT) under load: a load-invariant service floor (P10Low), the current median (P50), and the average in-flight requests at that median (inflightAtP50).Why
Routing decisions need to know how a model behaves under its current queue, not just a static latency number. These three values let a scorer predict the TTFT of the next request at the current in-flight level.
How
Each response's measured TTFT is paired with the queue depth it saw at dispatch (correlated request -> response via
CycleState) and stored in a fixed-size sliding window. Percentiles are recomputed periodically and published to the model attribute store for consumers.