diff --git a/.agents/skills/pr-agent-bench/SKILL.md b/.agents/skills/pr-agent-bench/SKILL.md new file mode 100644 index 00000000..cf8fdaf3 --- /dev/null +++ b/.agents/skills/pr-agent-bench/SKILL.md @@ -0,0 +1,209 @@ +--- +name: pr-agent-bench +description: >- + How to read and interpret an aws-blocks PR agent-bench run β€” what the bench report's columns, + colored balls (🟒 βšͺ 🟑 πŸ”΄), Judge dimensions (F/S/P/C/B), composite/Score, and stop_reason values + mean, plus a 60-second regression-triage runbook (top-line verdict β†’ red cells β†’ stop_reason triage + β†’ failure class β†’ trace). Use when reading or interpreting aws-blocks PR agent-bench results, + bench report columns/colors, deciding whether a PR regressed, or triaging a red/amber bench cell. +--- + +# Reading a PR agent-bench run + +## North star β€” the one question + +The bench exists to answer **one question per PR**: *did this change keep things steady-or-better, or +did it regress?* Every column, color, and number below is in service of answering that in seconds. A +healthy PR should be **dismissable in under 10 seconds** from the top-line alone; a genuine regression +should hand you a **failure class and a trace path** without a second tool. + +> ⚠️ **Reading model vs. what's live today.** This runbook describes the *target* reading model the +> bench is being built toward. A few mechanics in it β€” the per-cell **noise band** recolor, the single +> **PR-verdict banner**, folded deep-dive, and per-red trace links β€” are **planned (v2), not yet on this +> branch**. Where a mechanic isn't live yet, the runbook tells you how to derive the same judgment from +> what the current (v1) report *does* show. See **[Implementation status](#implementation-status-as-of-pr-194)** +> for the exact live-vs-planned split. Never assume a v2 signal is present just because it's described here. + +## Color semantics + +Each cell's color is keyed off that cell's **noise band** β€” its historical run-to-run variance β€” not a +fixed number. The band is what makes a signal trustworthy at N=1. + +| Ball | Meaning | Action | +|---|---|---| +| 🟒 green | Improved **beyond** the cell's band | Glance only β€” good news, no action | +| βšͺ gray | **Steady within band**, OR new / no baseline | Nothing to see β€” expected noise or a first observation | +| 🟑 amber | Moved the **wrong way but within band** | **Watch, don't act** β€” one sample of normal variance | +| πŸ”΄ red | Composite Ξ” **≀ βˆ’5 AND** beyond band | **ACT** β€” a material, real regression | + +The whole point of the band is to stop 🟑 normal-variance swings from masquerading as πŸ”΄ regressions, +while still catching a small-but-real drop on an otherwise rock-steady cell. + +> **Today (v1)** the table does *not* yet use per-cell bands. It colors each metric by its delta vs the +> baseline using **fixed thresholds**: 🟒 beyond +threshold, πŸ”΄ beyond βˆ’threshold, 🟑 within Β±threshold +> (either direction = noise), βšͺ no baseline (tagged `(new)`), πŸ—‘οΈ a cell that existed in the baseline but +> is gone now. Thresholds: composite Β±5, Score Β±5, Judge Β±0.3, Tests Β±1 pass, +> Cost Β±max($0.02, 10% of baseline), Turns Β±3. Consequence: a big drop on a naturally-swingy cell colors +> πŸ”΄ under v1 even when the band would call it amber (see the worked example). + +## PR verdict (top-line) + +The run rolls up to one of three verdicts: + +- **REGRESSED** β€” PR composite Ξ” is beyond the **βˆ’band**, OR **any** scenario is πŸ”΄. +- **IMPROVED** β€” PR composite Ξ” is beyond the **+band** AND there are **zero** πŸ”΄. +- **STEADY** β€” anything else (the common, healthy case). + +**STEADY or IMPROVED with 0 red β†’ ship on bench grounds.** You're done in one glance. + +> **Today (v1)** there is no single banner word. Derive the verdict yourself from the preword bullets: +> the *"Mean composite X/100 β€” 🟒/🟑/πŸ”΄ Ξ” vs `main`"* line (its ball uses a Β±5 composite band) plus the +> count of πŸ”΄ / 🟑 cells in the table. The per-cell `verdict` tally in the preword +> (`pass Β· partial Β· fail Β· harness_error`) is a *different* field β€” see [reference](#reference-what-the-current-v1-report-contains). + +## How to read a run in 60 seconds + +**Step 0 β€” Top-line only.** Read the verdict + composite Ξ” + counts of πŸ”΄ / 🟑. If **STEADY / IMPROVED +and 0 red β†’ stop here and ship** (on bench grounds). Most PRs end at Step 0. + +**Step 1 β€” If red, read each red's structured line.** For every πŸ”΄ cell, pull its one line: composite Ξ”, +worst Judge dimension, `stop_reason`, and trace path. Do not open anything yet. + +**Step 2 β€” Triage the cell's CLASS FIRST (before acting on the score).** A 0 means different things +depending on the cell's class, and only *one* class is excluded noise. Match the red cell to one of three +buckets β€” this mirrors the mean-inclusion rule and the `pass Β· partial Β· fail Β· harness_error` tally in the +[Reference](#reference-what-the-current-v1-report-contains): + +| Class | Signal (`stop_reason` / `klass_reason` / status) | In the headline mean? | What its 0 means β†’ action | +|---|---|---|---| +| **`harness_error`** | `cancelled` (CI abort); a pre-grade failure (`preflight_failed`, `oidc_failed`, `init_abort`); an **ungraceful** step-2 death under active isolation (surfaces as `in_progress` / no terminal stop_reason) | **EXCLUDED** from the mean | The harness or CI died before the agent's work could be judged. **NOT a PR signal β†’ RE-RUN the cell.** | +| **`agent_fail` / `dead_server`** | `agent_fail`: graceful `wall_clock_timeout` (ran out of its time budget), `max_tokens` / `error` (hit its output-token budget), `agent_timeout`. `dead_server`: `dev_server_dead` (app built but its dev-server never served / crashed) | **INCLUDED as composite 0 β€” this 0 moved the mean** | A **genuine, counted failure** β€” *not* excluded noise. The `stop_reason` tells you whether a **re-run** might recover it (a transient budget / throttle blip) or whether it's a persistent agent budget / quality regression. Re-run to disambiguate, but it counts as a real data point until it does. | +| **`scored`** | `end_turn` / `tool_use` / `stop_sequence` **with a dropped composite** | INCLUDED β€” graded on its tests | The agent finished and the result genuinely got worse β†’ **real agent-quality regression β†’ dig** (Step 3). | + +**The trap to avoid:** only **`harness_error`** is excluded / "not a PR signal." `agent_fail` (incl. `max_tokens`, +`agent_timeout`, graceful `wall_clock_timeout`) and `dead_server` are **counted composite-0 failures** that +already dragged the headline mean β€” a re-run tells you *transient vs. real*, but never wave them off as +"infra noise." + +**Step 3 β€” Worst dimension β†’ failure class β†’ where to look.** The Judge's lowest dimension points you +straight at the code. The table's Judge column shows only the **averaged** score β€” the per-dimension +scores (F/S/P/C/B) are **not** in the at-a-glance row; read them from the cell's judge artifact JSON +(`bench-result--