Skip to content

Commit f801dcf

Browse files
halleriteclaude
andcommitted
docs(v1): evaluate-environments skill matches the episode/agent contract
- durability: traces.jsonl is one EPISODE per line, appended after the episode finishes (whole or not at all) — not per-trace - timeouts: TaskData.timeout is overridden by the AGENT's TimeoutConfig (--env.<agent>.timeout.*), not a run-level one RolloutLimits wording was already agent-scoped; no change there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 1f18bf0 commit f801dcf

2 files changed

Lines changed: 12 additions & 11 deletions

File tree

skills/evaluate-environments/SKILL.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ outputs/<env>--<model>--<harness>/<uuid>/
160160
└── eval.log
161161
```
162162

163-
Set an exact path with `-o`. Results append as each trace finishes.
163+
Set an exact path with `-o`. `traces.jsonl` is one **episode** per line — an env-rollout's traces plus their shared standing — appended after each episode finishes, so an episode is durable whole or not at all (a torn last line is the whole episode redone on resume).
164164

165165
Resume in place:
166166

skills/evaluate-environments/references/REFERENCE.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -430,16 +430,16 @@ Unsupported fields are ignored; evaluation warns once per runtime/field combinat
430430

431431
### `TaskTimeout` (frozen)
432432
Per-row wall-clock timeout requests, in seconds, one for each rollout stage. For eval, a non-`None`
433-
value in the run's `TimeoutConfig` wins; otherwise the corresponding row value is used. If both are
434-
`None`, that stage has no framework timeout. Remote harness execution is capped at the provider's
435-
24-hour sandbox lifetime.
433+
value in the agent's `TimeoutConfig` (`--env.<agent>.timeout.*`) wins; otherwise the corresponding
434+
row value is used. If both are `None`, that stage has no framework timeout. Remote harness execution
435+
is capped at the provider's 24-hour sandbox lifetime.
436436

437437
| Field | Type | Default | Notes |
438438
|---|---|---|---|
439-
| `setup` | `float \| None` | `None` | Task and harness setup stage. Overridden by eval's `timeout.setup`; validate uses it for `Task.setup` when `CheckTimeoutConfig.setup` is unset. |
440-
| `harness` | `float \| None` | `None` | Harness execution. Overridden by the run-level `timeout.rollout`. |
441-
| `finalize` | `float \| None` | `None` | Task `finalize` hook. Overridden by `timeout.finalize`. |
442-
| `scoring` | `float \| None` | `None` | Task rewards/metrics/judges and harness metrics. Overridden by `timeout.scoring`. |
439+
| `setup` | `float \| None` | `None` | Task and harness setup stage. Overridden by the agent's `timeout.setup`; validate uses it for `Task.setup` when `CheckTimeoutConfig.setup` is unset. |
440+
| `harness` | `float \| None` | `None` | Harness execution. Overridden by the agent's `timeout.rollout`. |
441+
| `finalize` | `float \| None` | `None` | Task `finalize` hook. Overridden by the agent's `timeout.finalize`. |
442+
| `scoring` | `float \| None` | `None` | Task rewards/metrics/judges and harness metrics. Overridden by the agent's `timeout.scoring`. |
443443

444444
### `TaskData` (frozen)
445445
| Field | Type | Default | Notes |
@@ -649,9 +649,10 @@ log line per task.
649649
`TaskData`; otherwise a non-`None` row value fills it, and otherwise the runtime/provider default
650650
remains. `TaskData.image` is the required image for that row and replaces the runtime's base
651651
image. Unsupported resource fields are ignored; evaluation warns once per runtime/field.
652-
- **Timeout precedence.** For eval stages, a non-`None` run-level `TimeoutConfig` value wins over
653-
the corresponding `TaskData.timeout` value; if both are `None`, there is no framework timeout.
654-
The setup value is one deadline shared by task setup and harness provisioning.
652+
- **Timeout precedence.** For eval stages, a non-`None` agent-level `TimeoutConfig` value
653+
(`--env.<agent>.timeout.*`) wins over the corresponding `TaskData.timeout` value; if both are
654+
`None`, there is no framework timeout. The setup value is one deadline shared by task setup and
655+
harness provisioning.
655656
Validate uses `CheckTimeoutConfig.setup`, then falls back to `TaskData.timeout.setup`, while
656657
`CheckTimeoutConfig.total` independently bounds `Task.validate`.
657658
- **Discriminated unions** are selected by their `type` field: `client.type` (eval|train), `pool.type` (static|elastic), `env.<agent>.harness.runtime.type` / `runtime.type` (subprocess|docker|prime|modal).

0 commit comments

Comments
 (0)