diff --git a/.agents/skills/pr-agent-bench/SKILL.md b/.agents/skills/pr-agent-bench/SKILL.md new file mode 100644 index 000000000..14f5c672e --- /dev/null +++ b/.agents/skills/pr-agent-bench/SKILL.md @@ -0,0 +1,187 @@ +--- +name: pr-agent-bench +description: >- + How to read and interpret an aws-blocks PR agent-bench run β€” what the bench report's columns, + colored balls (🟒 βšͺ 🟑 πŸ”΄), Judge dimensions (F/S/P/C/B), composite/Score, and stop_reason values + mean, plus a 60-second regression-triage runbook (top-line verdict β†’ red cells β†’ stop_reason triage + β†’ failure class β†’ trace). Use when reading or interpreting aws-blocks PR agent-bench results, + bench report columns/colors, deciding whether a PR regressed, or triaging a red/amber bench cell. +--- + +# Reading a PR agent-bench run + +## North star β€” the one question + +The bench exists to answer **one question per PR**: *did this change keep things steady-or-better, or +did it regress?* Every column, color, and number below is in service of answering that in seconds. A +healthy PR should be **dismissable in under 10 seconds** from the top-line alone; a genuine regression +should hand you a **failure class and a trace path** without a second tool. + +> ⚠️ **Reading model vs. what's live today.** This runbook describes the *target* reading model the +> bench is being built toward. A few mechanics in it β€” the per-cell **noise band** recolor, the single +> **PR-verdict banner**, folded deep-dive, and per-red trace links β€” are **planned (v2), not yet on this +> branch**. Where a mechanic isn't live yet, the runbook tells you how to derive the same judgment from +> what the current (v1) report *does* show. See **[Implementation status](#implementation-status-as-of-pr-194)** +> for the exact live-vs-planned split. Never assume a v2 signal is present just because it's described here. + +## Color semantics + +Each cell's color is keyed off that cell's **noise band** β€” its historical run-to-run variance β€” not a +fixed number. The band is what makes a signal trustworthy at N=1. + +| Ball | Meaning | Action | +|---|---|---| +| 🟒 green | Improved **beyond** the cell's band | Glance only β€” good news, no action | +| βšͺ gray | **Steady within band**, OR new / no baseline | Nothing to see β€” expected noise or a first observation | +| 🟑 amber | Moved the **wrong way but within band** | **Watch, don't act** β€” one sample of normal variance | +| πŸ”΄ red | Composite Ξ” **≀ βˆ’5 AND** beyond band | **ACT** β€” a material, real regression | + +The whole point of the band is to stop 🟑 normal-variance swings from masquerading as πŸ”΄ regressions, +while still catching a small-but-real drop on an otherwise rock-steady cell. + +> **Today (v1)** the table does *not* yet use per-cell bands. It colors each metric by its delta vs the +> baseline using **fixed thresholds**: 🟒 beyond +threshold, πŸ”΄ beyond βˆ’threshold, 🟑 within Β±threshold +> (either direction = noise), βšͺ no baseline (tagged `(new)`), πŸ—‘οΈ a cell that existed in the baseline but +> is gone now. Thresholds: composite Β±5, Score Β±5, Judge Β±0.3 (also per-dimension), Tests Β±1 pass, +> Cost Β±max($0.02, 10% of baseline), Turns Β±3. Consequence: a big drop on a naturally-swingy cell colors +> πŸ”΄ under v1 even when the band would call it amber (see the worked example). + +## PR verdict (top-line) + +The run rolls up to one of three verdicts: + +- **REGRESSED** β€” PR composite Ξ” is beyond the **βˆ’band**, OR **any** scenario is πŸ”΄. +- **IMPROVED** β€” PR composite Ξ” is beyond the **+band** AND there are **zero** πŸ”΄. +- **STEADY** β€” anything else (the common, healthy case). + +**STEADY or IMPROVED with 0 red β†’ ship on bench grounds.** You're done in one glance. + +> **Today (v1)** there is no single banner word. Derive the verdict yourself from the preword bullets: +> the *"Mean composite X/100 β€” 🟒/🟑/πŸ”΄ Ξ” vs `main`"* line (its ball uses a Β±5 composite band) plus the +> count of πŸ”΄ / 🟑 cells in the table. The per-cell `verdict` tally in the preword +> (`pass Β· partial Β· fail Β· harness_error`) is a *different* field β€” see [reference](#reference-what-the-current-v1-report-contains). + +## How to read a run in 60 seconds + +**Step 0 β€” Top-line only.** Read the verdict + composite Ξ” + counts of πŸ”΄ / 🟑. If **STEADY / IMPROVED +and 0 red β†’ stop here and ship** (on bench grounds). Most PRs end at Step 0. + +**Step 1 β€” If red, read each red's structured line.** For every πŸ”΄ cell, pull its one line: composite Ξ”, +worst Judge dimension, `stop_reason`, and trace path. Do not open anything yet. + +**Step 2 β€” Triage `stop_reason` FIRST (before believing the score).** The score is only meaningful if the +run actually finished. + +| stop_reason class | Values | Verdict | +|---|---|---| +| Infra / harness noise | `dead_server` (`dev_server_dead`), harness errors (`preflight_failed`, `oidc_failed`, `init_abort`, `cancelled`, `wall_clock_timeout`), `max_tokens`, `agent_timeout` | **NOT a PR signal β†’ RE-RUN the cell.** A 0 here means the harness/agent died, not that the PR broke anything. | +| Real completion | `end_turn` **with a dropped composite** | **Real agent-quality regression β†’ dig** (go to Step 3). The agent finished and the result genuinely got worse. | + +**Step 3 β€” Worst dimension β†’ failure class β†’ where to look.** The Judge's lowest dimension points you +straight at the code: + +| Dimension ↓ (letter) | Failure class | Where to look | +|---|---|---| +| **functional_completeness** (F) | Task not done | The **feature code the PR touched** | +| **selector_contract** (S) | Selectors broke | **DOM / UI contract drift** (renamed ids, moved elements) | +| **persistence** (P) | State didn't survive | **State / storage / auth round-trip** broken | +| **code_quality** (C) | Quality regressed | The generated code got worse | +| **blocks_fidelity** (B) | Stopped using the framework idiomatically | **KEY signal for framework PRs** β€” the change nudged the agent away from Blocks idioms | + +**Step 4 β€” Open the trace, confirm root cause, decide.** Open the cell's trace at its +[stable artifact path](#finding-the-trace-and-per-cell-artifacts), confirm the root cause, then **fix vs +revert**. + +**Step 5 β€” Amber is a watch-list, not a blocker.** 🟑 cells are non-blocking on their own. But if the +**same cell** goes amber across **consecutive PRs**, that's a **slow regression** creeping in under the +band β€” investigate it proactively. + +## Materiality rule + +**πŸ”΄ requires BOTH conditions: composite Ξ” ≀ βˆ’5 AND beyond the cell's noise band.** + +- The band kills **variance false-alarms** (a swingy cell dropping within its normal range is amber, not red). +- The **βˆ’5 floor** is a materiality gate: it catches a real, small regression on an otherwise steady cell, + and refuses to raise the alarm for a sub-threshold wobble. βˆ’5 composite points is the minimum drop worth + a human's attention. + +## Worked example β€” run `29464250153` (a real messy run) + +This run shows why Step 2 (`stop_reason` first) and the noise band exist: + +- **`cognito` scored 0** β€” but via a **harness / `max_tokens` stop**, not a real failure. Per Step 2 this + is **infra β†’ re-run**. A later re-run of that cell scored **92**. A reader who trusted the 0 without + triaging the stop reason would have chased a phantom regression. +- **`auth-notes` posted a βˆ’78.8 composite delta** β€” alarming at first glance, and **v1's fixed Β±5 + threshold colors it πŸ”΄**. But that cell's **historical band is 11.6–94.4**: its composite naturally + swings across nearly the whole range, so βˆ’78.8 lands **within band β†’ 🟑 amber / watch, NOT a red + ship-blocker.** This is the exact case the band is designed to reclassify. + +Takeaway: **`stop_reason`-triage + the noise band together prevent phantom-regression chases.** Without +them, this run reads as two regressions; with them, it reads as "one infra re-run, one watch-list amber." + +## Finding the trace and per-cell artifacts + +Each scenario cell writes three GitHub Actions artifacts, named `--