You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(v1): evaluate-environments skill matches the episode/agent contract
- durability: traces.jsonl is one EPISODE per line, appended after the
episode finishes (whole or not at all) — not per-trace
- timeouts: TaskData.timeout is overridden by the AGENT's TimeoutConfig
(--env.<agent>.timeout.*), not a run-level one
RolloutLimits wording was already agent-scoped; no change there.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Set an exact path with `-o`. Results append as each trace finishes.
163
+
Set an exact path with `-o`. `traces.jsonl` is one **episode** per line — an env-rollout's traces plus their shared standing — appended after each episode finishes, so an episode is durable whole or not at all (a torn last line is the whole episode redone on resume).
Copy file name to clipboardExpand all lines: skills/evaluate-environments/references/REFERENCE.md
+11-10Lines changed: 11 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -430,16 +430,16 @@ Unsupported fields are ignored; evaluation warns once per runtime/field combinat
430
430
431
431
### `TaskTimeout` (frozen)
432
432
Per-row wall-clock timeout requests, in seconds, one for each rollout stage. For eval, a non-`None`
433
-
value in the run's `TimeoutConfig` wins; otherwise the corresponding row value is used. If both are
434
-
`None`, that stage has no framework timeout. Remote harness execution is capped at the provider's
435
-
24-hour sandbox lifetime.
433
+
value in the agent's `TimeoutConfig`(`--env.<agent>.timeout.*`) wins; otherwise the corresponding
434
+
row value is used. If both are `None`, that stage has no framework timeout. Remote harness execution
435
+
is capped at the provider's 24-hour sandbox lifetime.
436
436
437
437
| Field | Type | Default | Notes |
438
438
|---|---|---|---|
439
-
|`setup`|`float \| None`|`None`| Task and harness setup stage. Overridden by eval's `timeout.setup`; validate uses it for `Task.setup` when `CheckTimeoutConfig.setup` is unset. |
440
-
|`harness`|`float \| None`|`None`| Harness execution. Overridden by the run-level`timeout.rollout`. |
441
-
|`finalize`|`float \| None`|`None`| Task `finalize` hook. Overridden by `timeout.finalize`. |
442
-
|`scoring`|`float \| None`|`None`| Task rewards/metrics/judges and harness metrics. Overridden by `timeout.scoring`. |
439
+
|`setup`|`float \| None`|`None`| Task and harness setup stage. Overridden by the agent's `timeout.setup`; validate uses it for `Task.setup` when `CheckTimeoutConfig.setup` is unset. |
440
+
|`harness`|`float \| None`|`None`| Harness execution. Overridden by the agent's`timeout.rollout`. |
441
+
|`finalize`|`float \| None`|`None`| Task `finalize` hook. Overridden by the agent's `timeout.finalize`. |
442
+
|`scoring`|`float \| None`|`None`| Task rewards/metrics/judges and harness metrics. Overridden by the agent's `timeout.scoring`. |
443
443
444
444
### `TaskData` (frozen)
445
445
| Field | Type | Default | Notes |
@@ -649,9 +649,10 @@ log line per task.
649
649
`TaskData`; otherwise a non-`None` row value fills it, and otherwise the runtime/provider default
650
650
remains. `TaskData.image` is the required image for that row and replaces the runtime's base
651
651
image. Unsupported resource fields are ignored; evaluation warns once per runtime/field.
652
-
-**Timeout precedence.** For eval stages, a non-`None` run-level `TimeoutConfig` value wins over
653
-
the corresponding `TaskData.timeout` value; if both are `None`, there is no framework timeout.
654
-
The setup value is one deadline shared by task setup and harness provisioning.
652
+
-**Timeout precedence.** For eval stages, a non-`None` agent-level `TimeoutConfig` value
653
+
(`--env.<agent>.timeout.*`) wins over the corresponding `TaskData.timeout` value; if both are
654
+
`None`, there is no framework timeout. The setup value is one deadline shared by task setup and
655
+
harness provisioning.
655
656
Validate uses `CheckTimeoutConfig.setup`, then falls back to `TaskData.timeout.setup`, while
0 commit comments