|
| 1 | +# experiments/ |
| 2 | + |
| 3 | +Lab notes and matrices for the interactive-tmux workspace provider (EXP-01..06, |
| 4 | +FRICTION-*, ANALYTICS). See `INDEX.md` for the full map. |
| 5 | + |
| 6 | +## Acceptance harness: the Rust `itmux run` live-eval |
| 7 | + |
| 8 | +The acceptance battery that proves the `itmux run` contract end-to-end is the |
| 9 | +**Rust live-eval**, not the old Python `standalone_eval.py` (retired - it drove |
| 10 | +the superseded Python orchestrator on PR #240). |
| 11 | + |
| 12 | +- Harness: `providers/workspaces/interactive-tmux/driver-rs/tests/live_eval.rs` |
| 13 | +- Run it: `just eval-live` |
| 14 | + (= `AGENTIC_LIVE_EVAL=1 cargo test --test live_eval -- --ignored --test-threads=1 --nocapture`) |
| 15 | + |
| 16 | +The live cases (E1-E7) are BOTH `#[ignore]` AND env-gated on `AGENTIC_LIVE_EVAL=1`, |
| 17 | +so a plain `cargo test` never touches docker. The harness's plumbing (argv |
| 18 | +builder, JSONL event parser, `AgentRunResult` extractor, and the R5 |
| 19 | +event-contract validator) is unit-tested in that same file with no docker. |
| 20 | + |
| 21 | +### Auth prerequisite (live only) |
| 22 | + |
| 23 | +`itmux` sources host credentials into each workspace: |
| 24 | + |
| 25 | +- Claude: `~/.claude/.credentials.json` |
| 26 | +- Codex: `~/.codex/auth.json` |
| 27 | + |
| 28 | +If the claude token has expired, refresh it with `claude setup-token`. Without |
| 29 | +valid credentials, E1/E2 fail with an auth banner and the harness correctly |
| 30 | +reports `success=false` (the run still terminalizes cleanly with no orphan). |
| 31 | + |
| 32 | +### What the battery proves |
| 33 | + |
| 34 | +| Case | Proves | |
| 35 | +|------|--------| |
| 36 | +| E1 | claude happy-path: `result.success` + `EXPERIMENT_OK` in `session_log` + R5 contract | |
| 37 | +| E2 | codex reaches ready + graceful terminal (task success is a stretch goal) | |
| 38 | +| E3 | events arrive incrementally (arrival spread), `tool_start` precedes `session_end` | |
| 39 | +| E4 | graceful (1x SIGINT) / hard (2x SIGINT or SIGTERM) cancel at the submit/await boundary maps to the right terminal reason; orphan sweep clean | |
| 40 | +| E5b | bad image fails with no container created; orphan sweep clean | |
| 41 | +| E6 | short `--timeout` yields a `timeout` terminal reason, `success=false`, no orphan | |
| 42 | +| E7 | `--result-file` lands the result JSON in the file AND keeps stdout pure lifecycle events | |
| 43 | + |
| 44 | +E5a (forced startup-timeout orphan guard) is **unit-proven**, not live-proven |
| 45 | +(plan R2): `driver-rs/tests/orchestrator.rs` covers the teardown-on-startup- |
| 46 | +failure and hard-cancel-no-handle reap paths deterministically without docker |
| 47 | +(`start_failure_after_partial_startup_tears_down_and_no_orphan`, |
| 48 | +`hard_cancel_before_provision_reaps_orphans_best_effort`). A live forced timeout |
| 49 | +would re-test docker, not our logic. |
| 50 | + |
| 51 | +### Known gaps (unproven by this battery, plan R7) |
| 52 | + |
| 53 | +These `itmux run` surfaces are NOT exercised by E1-E7 and remain to be proven |
| 54 | +by later work: |
| 55 | + |
| 56 | +- **Inline credential materialization** - the battery uses host credentials |
| 57 | + (`~/.claude`, `~/.codex`); `AgentRunSpec.credentials` inline token/`auth.json` |
| 58 | + contents are not exercised. |
| 59 | +- **`--json false`** - only the default JSON event stream is validated; the |
| 60 | + human-summary mode is not. |
| 61 | +- **Output artifacts** - `AgentRunResult.output_artifacts` is always empty in |
| 62 | + v1; artifact collection is unproven. |
| 63 | +- **Observability** - `AgentRunSpec.observability` exporters and |
| 64 | + `AgentRunResult.observability` are Plan 3 placeholders, untested here. |
| 65 | +- **Token usage** - `token_usage` events / `AgentRunLimits.token_budget` |
| 66 | + enforcement are not asserted. |
| 67 | +- **Bundled-skill staging failure (#249)** - the fail-fast on host-path skill |
| 68 | + refs is unit-covered, not exercised live. |
| 69 | +- **Subagents** - v1 executes only the `default_agent`; declared `subagents` |
| 70 | + are validated-only (R5) and never spawned. |
0 commit comments