fix(agentx): enforce trace idle caps at runtime only

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

Author: cquil11

docs/benchmark-modes/semianalysis-agentx-faq.md

@@ -66,8 +66,8 @@ transparency.
   is rejected), and a fresh `--random-seed` is filled in — pin one yourself for reproducible run-to-run
   comparisons ([§7](#7-reading-the-results-metrics-validity-and-submission-requirements)). Timing mode is
   locked to agentic-replay.
-- **Optional timing control:** `--trace-idle-gap-cap-seconds` compresses long
-  request-start gaps independently within each root trace.
+- **Optional timing control:** `--trace-idle-gap-cap-seconds` limits observed
+  whole-tree idle time independently for each root and all of its descendants.
 - **Forbidden — do not pass:** `--ignore-trace-delays`,
   `--inter-turn-delay-cap-seconds`, `--synthesis-max-isl` (input truncation), and the
   rate/schedule flags `--request-rate` / `--arrival-pattern` / `--user-centric-rate` / `--fixed-schedule`
@@ -285,8 +285,9 @@ request-start to request-start. This is always the case for weka trace replay; t
 change it. Replay dispatches each turn only after the previous one completes,
 so start-to-start deltas would double-count the server's own response time and make every session
 drift later turn by turn. Individual trace gaps are not capped by default; pass
-`--trace-idle-gap-cap-seconds S` to compress request-start gaps longer than `S` independently
-within each root trace. If the entire replay has no active or ready request, AIPerf uniformly
+`--trace-idle-gap-cap-seconds S` to advance a trajectory's pending timers when its root and
+all descendants have had no request in flight for `S` seconds. This runtime guard does not
+rewrite dataset timestamps or bypass spawn/join dependencies. If the entire replay has no active or ready request, AIPerf uniformly
 shifts every pending request timer so the next request arrives within 10 seconds.
 
 ### Q: What does a single session look like on the wire?
@@ -795,7 +796,7 @@ the context-overflow bound is evaluated during the run.
 | SemiAnalysis corpus | A pinned `*_weka_*` `--public-dataset` alias, or `weka_hf` pinned to `semianalysisai/cc-traces-weka-062126` | A non-pinned dataset, local `weka_trace`, or another `--hf-weka-dataset` under the scenario; omitting a dataset entirely (CLI synthetic default) | Explicit wrong / unpinned loader: refuses to start (or `submission_valid: false` via `--unsafe-override`). Missing/synthetic: always refuses — `--unsafe-override` cannot bypass |
 | `ignore_eos` | Injected `ignore_eos=true` | Explicit `ignore_eos=false` | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
 | Streaming | `--streaming` auto-enabled | Explicit `--no-streaming` | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
-| Honored trace delays | Recorded think-time gaps are preserved by default; `--trace-idle-gap-cap-seconds` may explicitly compress per-trace gaps | `--ignore-trace-delays` or `--inter-turn-delay-cap-seconds` | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
+| Honored trace delays | Recorded think-time gaps are preserved; `--trace-idle-gap-cap-seconds` may bound observed whole-tree runtime idle without rewriting them | `--ignore-trace-delays` or `--inter-turn-delay-cap-seconds` | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
 | No input truncation | Prompts built to the full recorded token counts | Truncating/capping input below the recorded length | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
 | Minimum duration | `--benchmark-duration` ≥ 900s (default 1800s) | A duration below 900s | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
 | First-turn-prefix cache-busting | Uniqueness marker placed as a first-turn prefix | Disabling cache-busting or relocating the marker | Refuses to start (or `submission_valid: false` via `--unsafe-override`) |
@@ -1154,7 +1155,7 @@ The ones a serving engineer is most likely to use:
 | `--max-context-length` | Drops whole traces whose peak prompt+output exceeds your server's window, so you don't get guaranteed mid-run overflows. Blunter than a `_256k` corpus (removes entire traces, not just over-limit turns). If it would drop everything, the run errors instead of silently emptying the dataset. |
 | `--trajectory-start-min-ratio` / `--trajectory-start-max-ratio` | The window within each session where t\* is sampled — i.e. how deep into sessions the measured traffic sits. Defaults to the **full session (0.0–1.0)** under the `inferencex-agentx-mvp` scenario; 0.25–0.75 is the generic CLI default when not scenario-locked. |
 | `--system-idle-gap-cap-seconds` | Caps only globally idle replay time (scenario default 10s); all pending timers shift uniformly, preserving order and relative spacing. |
-| `--trace-idle-gap-cap-seconds` | Optional per-trace request-start gap compression; unset by default and allowed by the scenario. |
+| `--trace-idle-gap-cap-seconds` | Optional observed whole-tree runtime idle cap; unset by default and allowed by the scenario. |
 | `--inter-turn-delay-cap-seconds` | Per-turn timing compression; forbidden by the scenario. |
 | `--cache-bust` | Where the per-session uniqueness marker goes; the scenario locks `first_turn_prefix`. |
 

docs/cli-options.md

@@ -1092,7 +1092,7 @@ AGENTIC_REPLAY only: collapse the WARMUP-start and PROFILING-start dispatches in
 
 #### `--trace-idle-gap-cap-seconds` `<float>`
 
-Hard ceiling (seconds) for idle gaps within each individual trace. For Weka trace replay, AIPerf looks at all parent and subagent request submission timestamps within one root trace, compresses long gaps between consecutive request submissions, and derives turn delays from the compressed per-trace timeline. Original request api_time values are not used to decide these idle gaps. During AgentX profiling, AIPerf also advances only that runtime tree's pending replay timers if independent stream clocks would leave the whole tree idle beyond the cap; request order and dependency gates remain unchanged. When set for Weka, this takes precedence over `--inter-turn-delay-cap-seconds` so individual parent/subagent-line delays are not separately capped. Defaults to None (no per-trace idle-gap compression).
+Hard ceiling (seconds) for observed runtime idle time within each AgentX trajectory tree. The idle clock covers initial future work and restarts when the final in-flight request across the root and all descendant streams completes. If no request from that tree reaches the wire before the cap, AIPerf uniformly advances only that tree's pending replay timers. Dataset timestamps are unchanged, and request order, spawn/join dependencies, and replay barriers remain authoritative. Defaults to None (no per-trace runtime idle cap).
 <br/>_Constraints: ≥ 0.0_
 
 #### `--system-idle-gap-cap-seconds` `<float>`
@@ -2627,7 +2627,7 @@ AGENTIC_REPLAY only: collapse the WARMUP-start and PROFILING-start dispatches in
 
 #### `--trace-idle-gap-cap-seconds` `<float>`
 
-Hard ceiling (seconds) for idle gaps within each individual trace. For Weka trace replay, AIPerf looks at all parent and subagent request submission timestamps within one root trace, compresses long gaps between consecutive request submissions, and derives turn delays from the compressed per-trace timeline. Original request api_time values are not used to decide these idle gaps. During AgentX profiling, AIPerf also advances only that runtime tree's pending replay timers if independent stream clocks would leave the whole tree idle beyond the cap; request order and dependency gates remain unchanged. When set for Weka, this takes precedence over `--inter-turn-delay-cap-seconds` so individual parent/subagent-line delays are not separately capped. Defaults to None (no per-trace idle-gap compression).
+Hard ceiling (seconds) for observed runtime idle time within each AgentX trajectory tree. The idle clock covers initial future work and restarts when the final in-flight request across the root and all descendant streams completes. If no request from that tree reaches the wire before the cap, AIPerf uniformly advances only that tree's pending replay timers. Dataset timestamps are unchanged, and request order, spawn/join dependencies, and replay barriers remain authoritative. Defaults to None (no per-trace runtime idle cap).
 <br/>_Constraints: ≥ 0.0_
 
 #### `--system-idle-gap-cap-seconds` `<float>`

docs/tutorials/agentx-mvp.md

@@ -267,7 +267,7 @@ flag.
 | `--streaming` is on | Responses stream token-by-token (auto-enabled when unset; explicit `--no-streaming` errors) | The per-token latency metrics (TTFT, ITL) are core to this benchmark and need streaming responses. |
 | Replay delays are end-to-start (always on, no flag) | Each turn's replay delay is the recorded idle gap from the previous response's *end* to the next request's *start*, not the start-to-start delta. This is unconditional for weka trace replay — there is no toggle. | Replay dispatches each turn after the previous one completes, so start-to-start deltas would double-count the previous request's server time, making every session drift later turn by turn and overstating how many sessions overlap at once. |
 | `--ignore-trace-delays` is off | Trace-derived delays are preserved | The replay retains the captured agent pacing and KV-cache reuse intervals. |
-| Per-trace idle-gap cap is optional | `--trace-idle-gap-cap-seconds` is unset by default and accepts an explicit CLI value | Setting it compresses long request-start gaps independently within each root trace, changing think-time and cache-TTL behavior while avoiding lanes held by long idle gaps. |
+| Per-trace idle-gap cap is optional | `--trace-idle-gap-cap-seconds` is unset by default and accepts an explicit CLI value | Setting it bounds observed runtime idle across the root and every descendant stream without rewriting dataset timestamps or bypassing spawn/join dependencies. |
 | Per-turn cap is forbidden | `--inter-turn-delay-cap-seconds` must remain unset | Independently clamping parent and subagent delays can distort their relative timing. |
 | `--system-idle-gap-cap-seconds = 10` | When no request is active or ready, all pending replay timers shift uniformly so the next request arrives within 10 seconds | The benchmark avoids measuring long periods with no server work while preserving request order and relative spacing across every pending trajectory. |
 | `--cache-bust first_turn_prefix` | A unique per-conversation marker is injected at the start of the first user turn for every play (each dispatch of a trace, initial or recycled) | Without this, every time a trace is recycled the server's prefix cache would warm up further on identical content, and steady-state cache-hit rates would inflate the longer the run goes. The marker gives every recycled play a fresh prompt prefix. |
@@ -440,10 +440,14 @@ request metrics.
 After warmup, the profiling phase opens. Now you're measuring. Each trajectory
 keeps replaying its conversation from turn `k_i + 1` onward, honoring the
 original recorded inter-turn gaps as end-to-start delays counted from the
-moment the previous turn completes. An explicit
-`--trace-idle-gap-cap-seconds S` first compresses request-start gaps longer
-than `S` independently within each root trace. If every request completes
-while future requests remain scheduled more than 10 seconds away, AIPerf
+moment the previous turn completes. With
+`--trace-idle-gap-cap-seconds S`, a per-trajectory watchdog covers initial
+future work and restarts when the last in-flight request across the root and
+all descendant streams completes.
+If that tree remains idle for `S` seconds, AIPerf uniformly advances only its
+pending timers; dataset timestamps, relative timer spacing, request order, and
+spawn/join gates remain unchanged. If every request completes while future
+requests remain scheduled more than 10 seconds away, AIPerf
 shifts all pending request timers earlier by the same amount. This bounds true
 system-idle time without changing request order or the relative spacing among
 pending trajectories. A scheduled turn may become retained by a cross-stream

docs/tutorials/weka-trace.md

@@ -104,7 +104,7 @@ The HuggingFace path and the file-based `--input-file` path produce **byte-ident
 | `--public-dataset semianalysis_cc_traces_weka` (HuggingFace, legacy/default no-subagents alias) | Legacy/default pinned alias for the same current no-subagents corpus as `semianalysis_cc_traces_weka_no_subagents`. |
 | `--public-dataset semianalysis_cc_traces_weka_with_subagents` (HuggingFace, with subagents) | Rolling with-subagents alias tracking the current default corpus (062126) for zero-setup runs with full subagent SPAWN/JOIN topology. 393 traces. |
 
-Existing Weka tunables work identically in both paths: `--synthesis-max-isl`, `--synthesis-max-osl`, `--inter-turn-delay-cap-seconds`, `--trace-idle-gap-cap-seconds`, `--ignore-trace-delays`, `--use-think-time-only`, `--cache-bust`, the per-trace model rewriting rules below — same flags, same behavior, same output bytes on the wire. For `--scenario inferencex-agentx-mvp`, the validator accepts a with-subagents `--public-dataset` alias or `weka_hf` constrained to `semianalysisai/cc-traces-weka-062126` for `submission_valid: true`; a local `weka_trace` directory requires `--unsafe-override` (stamps `submission_valid: false`) because corpus identity is not fingerprintable; it does not accept the no-subagents aliases. That scenario permits an explicit `--trace-idle-gap-cap-seconds`, forbids the per-turn `--inter-turn-delay-cap-seconds`, and retains the global `--system-idle-gap-cap-seconds` guard.
+Existing Weka tunables work identically in both paths: `--synthesis-max-isl`, `--synthesis-max-osl`, `--inter-turn-delay-cap-seconds`, `--trace-idle-gap-cap-seconds`, `--ignore-trace-delays`, `--use-think-time-only`, `--cache-bust`, the per-trace model rewriting rules below — same flags, same behavior, same output bytes on the wire. `--trace-idle-gap-cap-seconds` is a runtime AgentX watchdog and does not rewrite either loader's dataset timestamps. For `--scenario inferencex-agentx-mvp`, the validator accepts a with-subagents `--public-dataset` alias or `weka_hf` constrained to `semianalysisai/cc-traces-weka-062126` for `submission_valid: true`; a local `weka_trace` directory requires `--unsafe-override` (stamps `submission_valid: false`) because corpus identity is not fingerprintable; it does not accept the no-subagents aliases. That scenario permits an explicit `--trace-idle-gap-cap-seconds`, forbids the per-turn `--inter-turn-delay-cap-seconds`, and retains the global `--system-idle-gap-cap-seconds` guard.
 
 For newly published compatible HuggingFace Weka trace corpora, pass `--hf-weka-dataset <repo>` — it auto-selects the neutral `weka_hf` loader (passing `--public-dataset weka_hf` explicitly is equivalent):
 

src/aiperf/config/dataset/config.py

@@ -510,18 +510,14 @@ class FileDataset(BaseConfig):
         Field(
             default=None,
             ge=0.0,
-            description="Hard ceiling (seconds) for idle gaps within each individual trace. "
-            "For Weka trace replay, AIPerf looks at all parent and subagent request "
-            "submission timestamps within one root trace, compresses long gaps between "
-            "consecutive request submissions, and derives turn delays from the "
-            "compressed per-trace timeline. Original request api_time values are not "
-            "used to decide these idle gaps. During AgentX profiling, AIPerf also "
-            "advances only that runtime tree's pending replay timers if independent "
-            "stream clocks would leave the whole tree idle beyond the cap; request "
-            "order and dependency gates remain unchanged. When set for Weka, this "
-            "takes precedence over `--inter-turn-delay-cap-seconds` so individual "
-            "parent/subagent-line delays are not separately capped. Defaults to None "
-            "(no per-trace idle-gap compression).",
+            description="Hard ceiling (seconds) for observed runtime idle time within "
+            "each AgentX trajectory tree. The idle clock covers initial future work and "
+            "restarts when the final in-flight request across the root and all descendant "
+            "streams completes. If no request from that tree reaches the wire before the "
+            "cap, AIPerf uniformly advances "
+            "only that tree's pending replay timers. Dataset timestamps are unchanged, "
+            "and request order, spawn/join dependencies, and replay barriers remain "
+            "authoritative. Defaults to None (no per-trace runtime idle cap).",
         ),
     ]
 
@@ -797,12 +793,10 @@ class PublicDataset(BaseConfig):
         Field(
             default=None,
             ge=0.0,
-            description="Hard ceiling (seconds) for idle gaps within each "
-            "individual trace for HF-backed Weka replay (mirror of "
-            "``FileDataset.trace_idle_gap_cap_seconds``), including runtime "
-            "whole-tree idle caused by independent AgentX stream clocks. When "
-            "set, takes precedence over ``inter_turn_delay_cap_seconds``. "
-            "Defaults to None (no per-trace idle-gap compression).",
+            description="Hard ceiling (seconds) for observed runtime idle time within "
+            "each AgentX trajectory tree for HF-backed Weka replay (mirror of "
+            "``FileDataset.trace_idle_gap_cap_seconds``). Dataset timestamps remain "
+            "unchanged. Defaults to None (no per-trace runtime idle cap).",
         ),
     ]
 

src/aiperf/config/flags/_converter_dataset.py

@@ -808,11 +808,11 @@ def _apply_inter_turn_delay_cap(d: dict[str, Any], cli: CLIConfig) -> None:
 def _apply_trace_delay_flags(d: dict[str, Any], cli: CLIConfig) -> None:
     """Route trace-replay delay knobs onto ``FileDataset``/``PublicDataset``.
 
-    ``--ignore-trace-delays``, ``--use-think-time-only``, and
-    ``--trace-idle-gap-cap-seconds`` live on
-    both FILE and PUBLIC dataset models and bake into ``Turn.delay`` /
-    ``Turn.timestamp`` at load time. Without this route the CLI flags are
-    silently dropped (YAML / scenario paths set the fields directly).
+    These flags live on both FILE and PUBLIC dataset models. Ignore/think-only
+    affect ``Turn.delay`` / ``Turn.timestamp`` at load time; the trace idle cap
+    is consumed by the AgentX runtime watchdog without rewriting either field.
+    Without this route the CLI flags are silently dropped (YAML / scenario
+    paths set the fields directly).
     Mutual exclusivity of ignore vs think-only is enforced by the dataset
     model validators after conversion.
     """