Skip to content

Commit 51df883

Browse files
docs(itmux): eval acceptance harness note + just eval-live (#240)
Add a just eval-live wrapper (R3) that runs the gated live battery (AGENTIC_LIVE_EVAL=1 cargo test --test live_eval -- --ignored --test-threads=1 --nocapture), and an experiments/README pointing at the Rust live-eval as the acceptance harness. Documents the host-credential auth prereq (claude setup-token on expiry), the E1-E7 coverage table, E5a being unit-proven (R2), and the R7 unproven-by-this-battery surfaces (inline creds, --json false, artifacts, observability, token usage, bundled-skill staging #249, subagents). PR #240 (superseded Python orchestrator + standalone_eval.py) to be closed by the maintainer after this and the thin Python client land - not closed here. Claude-Session: https://claude.ai/code/session_019TtekhJft4s2qBGRGgxToG
1 parent 626c2b2 commit 51df883

2 files changed

Lines changed: 79 additions & 0 deletions

File tree

experiments/README.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# experiments/
2+
3+
Lab notes and matrices for the interactive-tmux workspace provider (EXP-01..06,
4+
FRICTION-*, ANALYTICS). See `INDEX.md` for the full map.
5+
6+
## Acceptance harness: the Rust `itmux run` live-eval
7+
8+
The acceptance battery that proves the `itmux run` contract end-to-end is the
9+
**Rust live-eval**, not the old Python `standalone_eval.py` (retired - it drove
10+
the superseded Python orchestrator on PR #240).
11+
12+
- Harness: `providers/workspaces/interactive-tmux/driver-rs/tests/live_eval.rs`
13+
- Run it: `just eval-live`
14+
(= `AGENTIC_LIVE_EVAL=1 cargo test --test live_eval -- --ignored --test-threads=1 --nocapture`)
15+
16+
The live cases (E1-E7) are BOTH `#[ignore]` AND env-gated on `AGENTIC_LIVE_EVAL=1`,
17+
so a plain `cargo test` never touches docker. The harness's plumbing (argv
18+
builder, JSONL event parser, `AgentRunResult` extractor, and the R5
19+
event-contract validator) is unit-tested in that same file with no docker.
20+
21+
### Auth prerequisite (live only)
22+
23+
`itmux` sources host credentials into each workspace:
24+
25+
- Claude: `~/.claude/.credentials.json`
26+
- Codex: `~/.codex/auth.json`
27+
28+
If the claude token has expired, refresh it with `claude setup-token`. Without
29+
valid credentials, E1/E2 fail with an auth banner and the harness correctly
30+
reports `success=false` (the run still terminalizes cleanly with no orphan).
31+
32+
### What the battery proves
33+
34+
| Case | Proves |
35+
|------|--------|
36+
| E1 | claude happy-path: `result.success` + `EXPERIMENT_OK` in `session_log` + R5 contract |
37+
| E2 | codex reaches ready + graceful terminal (task success is a stretch goal) |
38+
| E3 | events arrive incrementally (arrival spread), `tool_start` precedes `session_end` |
39+
| E4 | graceful (1x SIGINT) / hard (2x SIGINT or SIGTERM) cancel at the submit/await boundary maps to the right terminal reason; orphan sweep clean |
40+
| E5b | bad image fails with no container created; orphan sweep clean |
41+
| E6 | short `--timeout` yields a `timeout` terminal reason, `success=false`, no orphan |
42+
| E7 | `--result-file` lands the result JSON in the file AND keeps stdout pure lifecycle events |
43+
44+
E5a (forced startup-timeout orphan guard) is **unit-proven**, not live-proven
45+
(plan R2): `driver-rs/tests/orchestrator.rs` covers the teardown-on-startup-
46+
failure and hard-cancel-no-handle reap paths deterministically without docker
47+
(`start_failure_after_partial_startup_tears_down_and_no_orphan`,
48+
`hard_cancel_before_provision_reaps_orphans_best_effort`). A live forced timeout
49+
would re-test docker, not our logic.
50+
51+
### Known gaps (unproven by this battery, plan R7)
52+
53+
These `itmux run` surfaces are NOT exercised by E1-E7 and remain to be proven
54+
by later work:
55+
56+
- **Inline credential materialization** - the battery uses host credentials
57+
(`~/.claude`, `~/.codex`); `AgentRunSpec.credentials` inline token/`auth.json`
58+
contents are not exercised.
59+
- **`--json false`** - only the default JSON event stream is validated; the
60+
human-summary mode is not.
61+
- **Output artifacts** - `AgentRunResult.output_artifacts` is always empty in
62+
v1; artifact collection is unproven.
63+
- **Observability** - `AgentRunSpec.observability` exporters and
64+
`AgentRunResult.observability` are Plan 3 placeholders, untested here.
65+
- **Token usage** - `token_usage` events / `AgentRunLimits.token_budget`
66+
enforcement are not asserted.
67+
- **Bundled-skill staging failure (#249)** - the fail-fast on host-path skill
68+
refs is unit-covered, not exercised live.
69+
- **Subagents** - v1 executes only the `default_agent`; declared `subagents`
70+
are validated-only (R5) and never spawned.

justfile

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,15 @@ rust-fmt:
145145
cd providers/workspaces/interactive-tmux/driver-rs && cargo fmt
146146
@echo '{{ GREEN }}✓ itmux Rust formatting complete{{ NORMAL }}'
147147

148+
# Run the LIVE itmux run acceptance battery (E1-E7). Needs docker + valid host
149+
# credentials (~/.claude/.credentials.json, ~/.codex/auth.json). These cases
150+
# are gated out of `cargo test`; this is the only entry point that runs them.
151+
[group('rust')]
152+
eval-live:
153+
@echo '{{ YELLOW }}Running LIVE itmux run acceptance battery (docker + token required)...{{ NORMAL }}'
154+
cd providers/workspaces/interactive-tmux/driver-rs && AGENTIC_LIVE_EVAL=1 cargo test --test live_eval -- --ignored --test-threads=1 --nocapture
155+
@echo '{{ GREEN }}✓ itmux live acceptance battery complete{{ NORMAL }}'
156+
148157
# ═══════════════════════════════════════════════════════════════════════════════
149158
# COMBINED OPERATIONS
150159
# ═══════════════════════════════════════════════════════════════════════════════

0 commit comments

Comments
 (0)