Skip to content

Commit 770eb2f

Browse files
committed
feat(testing): let AI orchestrate the plan loop
1 parent ee24058 commit 770eb2f

3 files changed

Lines changed: 269 additions & 130 deletions

File tree

docs/testing/test-report-output-spec.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ This document defines the normative format for AI sessions, case summaries, run
88
<a id="report-principles"></a>
99
## 1. Principles
1010

11-
1. `run-plan` uses an AI scheduling session before each case, and each case it
12-
executes runs in an independent Codex or Claude session.
11+
1. `run-plan` uses one AI orchestration session to drive the `next-case` loop,
12+
and each case it executes runs in an independent Codex or Claude session.
1313
2. The agent's native JSONL is the primary source for commands, tool calls, and raw output.
1414
3. The agent writes one shallow `result.json`; it does not copy command output into that file.
1515
4. The runner generates `runner.json`, timestamps, exit code, checksums, and run aggregates.
@@ -52,7 +52,7 @@ The runner supplies the plan guide, `case.md`, output location, status rules, an
5252
For a dependency-driven skip, the orchestrator creates a synthetic one-event
5353
case session containing the reason and causal earlier case IDs. It does not
5454
pretend that the skipped case was executed. The full decision process remains
55-
available under the run-level `decisions/` directory.
55+
available in the run-level `orchestrator.jsonl`.
5656

5757
<a id="report-layout"></a>
5858
## 4. Result layout

tools/dstack-test/README.md

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -42,16 +42,15 @@ tools/dstack-test/dstack-test run-plan \
4242
-- "Follow the environment restrictions in README.md"
4343
```
4444

45-
Before each case, `run-plan` starts a scheduling-agent session that reads the
46-
guide, index, all case specifications, and prior results. The runner—not the
47-
scheduling agent—then invokes `run-case` in a fresh session when the decision is
48-
`RUN`. When a
49-
prior non-PASS result demonstrably invalidates a later prerequisite, it records
50-
that later case as `SKIPPED` with the causal case IDs instead of starting an
51-
unnecessary case session. It must not skip independent or merely expensive
52-
cases. The complete orchestration session is stored at
53-
`<plan>/results/<run-id>/decisions/`. Keeping scheduling and case execution in
54-
separate processes avoids nested agent CLIs and preserves trust boundaries.
45+
`run-plan` starts one orchestration-agent session. The agent drives the loop by
46+
calling `next-case`, followed by either `run-case` or `complete-case`, until the
47+
server reports `COMPLETE`. A file-based control channel lets the outer runner
48+
launch each case agent outside the orchestrator sandbox. When a prior non-PASS
49+
result demonstrably invalidates a later prerequisite, the orchestrator records
50+
that case as `SKIPPED`; it must not skip independent or merely expensive cases.
51+
The complete control conversation is stored in `orchestrator.jsonl`.
52+
53+
An interrupted run can be continued with the same run ID and `--resume`.
5554

5655
## Finalize and validate
5756

0 commit comments

Comments
 (0)