test(itmux): live acceptance battery over itmux run + --timeout passthrough (Plan B Task 8)#252
Open
NeuralEmpowerment wants to merge 6 commits into
Open
test(itmux): live acceptance battery over itmux run + --timeout passthrough (Plan B Task 8)#252NeuralEmpowerment wants to merge 6 commits into
NeuralEmpowerment wants to merge 6 commits into
Conversation
Add itmux run --timeout <secs> mapping to AgentRunSpec.limits.timeout_s via a pure build_run_spec helper. Additive: when omitted, limits stays None so the orchestrator's default await bound is unchanged. Proves the timeout terminal reason at the CLI (plan B Task 8, R6). Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
EXP-0005 recipe directories for the live acceptance battery (Plan B Task 8.1): recipe-claude-hello (claude, opus-4-8, effort low) and recipe-codex-hello (codex, gpt-5.5, effort low). Each SYSTEM prompt drives the agent to create /workspace/hello.txt and print the EXPERIMENT_OK sentinel so E1/E2 can verify task completion off the captured session_log. Also vendors a well-formed sample event stream (sample_run_good.jsonl) for the R5 contract-validator unit tests (no docker). Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
Rust live-eval harness driving the real itmux run binary end-to-end (Plan B Task 8.2/8.3), replacing the retired Python standalone_eval.py. Non-live plumbing (runs in cargo test, no docker): argv builder, JSONL event parser, AgentRunResult extractor (stream + --result-file), R5 event-contract validator (seq monotonic-from-0, one session_end, consistent run_id, result delivery channel, stdout purity), docker orphan sweep, PASS reporter. 17 unit tests cover the parser, argv builder, validator (good + corrupted streams), sentinel-from-session_log (R1), result-file round-trip, and eval-recipe load. Live cases E1-E7 are each #[ignore] AND early-return unless AGENTIC_LIVE_EVAL=1 (double safety): E1 claude happy-path (success + EXPERIMENT_OK in session_log), E2 codex graceful terminal, E3 incremental event arrival, E4 graceful/hard cancellation at the submit/await boundary (R4), E5b bad-image no-container, E6 timeout terminal reason (R6), E7 --result-file stdout purity (R7). E5a is unit-proven per R2 (orchestrator.rs orphan-guard tests), referenced in a comment, no live case added. R1 finding: AgentRunResult.session_log already carries the captured pane, so no contract field was added - E1 asserts the sentinel against it. Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
Add a just eval-live wrapper (R3) that runs the gated live battery (AGENTIC_LIVE_EVAL=1 cargo test --test live_eval -- --ignored --test-threads=1 --nocapture), and an experiments/README pointing at the Rust live-eval as the acceptance harness. Documents the host-credential auth prereq (claude setup-token on expiry), the E1-E7 coverage table, E5a being unit-proven (R2), and the R7 unproven-by-this-battery surfaces (inline creds, --json false, artifacts, observability, token usage, bundled-skill staging #249, subagents). PR #240 (superseded Python orchestrator + standalone_eval.py) to be closed by the maintainer after this and the thin Python client land - not closed here. Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
…undness) Fix 1 (codex must-fix): --timeout took an unvalidated f64, so --timeout=-1 / =0 / =NaN / =inf reached Duration::from_secs_f64 and panicked before docker. Add a clap value_parser (parse_positive_timeout) rejecting non-finite and non-positive values with a clean CLI error (exit 2, no panic). Unit tests cover -1/0/NaN/inf/garbage -> error and 2.5 -> Some(2.5). Fix 2 (codex must-fix): the event-contract validator dropped blank stdout lines at collection, so a blank line in --json mode slipped past the purity check. parse_lines no longer filters blank lines; a blank/whitespace-only line surfaces as NonJson and the purity rule rejects it. Negative test added. Fix 3 (codex must-fix): the validator counted exactly one session_end but did not enforce ordering, so session_end -> tool_start -> result (OnStream) and lifecycle events after session_end (ResultFile) wrongly passed. Enforce session_end as the terminal lifecycle event: only the OnStream result line may follow it, nothing else, in both modes. Negative tests added for both modes. Nits: removed the dead StreamedRun.exit_ok field (+ its allow(dead_code)); documented the E3 lexicographic-ts assumption (emitter always uses fixed-width second-precision UTC Z timestamps) rather than adding a datetime dependency. Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
Contributor
Author
Dual-review gate: PASS
Fixes: |
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Plan B Task 8 (codex plan-reviewed, all 7 revisions adopted). Replaces the old Python
standalone_eval.py(which drove the superseded Python orchestrator on #240) with a pure-Rust live acceptance battery driving theitmux runbinary end-to-end. Stacked on #247.What's in it
itmux run --timeout <secs>passthrough (R6) ->AgentRunSpec.limits.timeout_svia a purebuild_run_spechelper (unit-tested); omitted =>limits=None, orchestrator default (300s) unchanged. This proves thetimeoutterminal reason, previously unreachable from the CLI.tests/live_eval.rs- argv builder,AgentRunEventJSONL parser,AgentRunResultextractor, orphan-container sweep, and an event-contract validator (R5: monotonic seq from 0, exactly one session_end, consistent run_id, result viatype:"result"OR--result-file, zero non-JSON stdout lines).tests/fixtures/eval/recipe-{claude,codex}-hello/(EXP-0005 shape).cargo test, no docker): argv x3, parser x2, result+sentinel extraction, result-file round-trip, R5 validator x6, cancel-boundary detection, fixture-load x2,build_run_specx2.#[ignore]+ early-return unlessAGENTIC_LIVE_EVAL=1): E1 claude happy-path (assertsEXPERIMENT_OKinresult.session_logper R1, no new field needed - session_log already carries the pane), E2 codex, E3 live-events ordering, E4 graceful+hard cancel (R4: trigger at tool_end submit / tool_start await; assert terminal reason + no orphan), E5b bad-image orphan-guard, E6 timeout, E7 result-file purity. E5a is unit-proven (R2) via the orchestrator's partial-startup/no-orphan tests.just eval-live(R3) runs the gated battery.experiments/README.mddocuments the auth prereq (host creds,claude setup-tokenon expiry), the E1-E7 table, E5a-unit-proven, and the R7 unproven surfaces.Documented unproven-by-this-battery surfaces (R7)
inline credential materialization,
--json false, output_artifacts, observability bundle, token_usage/token_budget, bundled-skill staging (#249, unit-only), subagent execution (validated-only v1).Incidental finding -> tracked
Truncating
itmux runstdout (| head) SIGPIPE-kills the process before terminalize -> orphan container (uncovered by the SIGINT/SIGTERM two-tier cancel). Filed as #251 (fix alongside #248 teardown-by-name).Gates: cargo test all green (17 pass, 8 ignored), clippy -D warnings clean, fmt clean, R8 neutrality guard holds. Tracks okrs-o78. The live battery runs green once a valid claude/codex host token is present.