feat(bench): reshape tasks to api-first framework-focus grading

.changeset/bench-tasks-framework-focus.md

@@ -0,0 +1,20 @@
+---
+---
+
+test(bench): reshape agent-bench task graders to be framework-surface-first
+
+Internal CI/bench tooling only — changes are confined to the private, unpublished
+`@aws-blocks/agent-bench` workspace and its task content (`tasks/*`). No
+published-package (`packages/*`) changes, so this needs no release (empty changeset).
+
+- The per-task Playwright graders now assert the **framework `api` surface** (the
+  JSON-RPC `POST /aws-blocks/api` namespace) as the canonical signal, with a thin
+  page smoke on top, so a task is graded on genuine Building-Block usage rather than
+  DOM-only behavior that a mock UI could fake.
+- Follow-up review hardening (same PR): removed a dead, never-executed OTP-gate
+  `else` assertion in `cognito-profile` (the bench always runs with `BLOCKS_MOCK=true`)
+  and documented that the production gate is prompt-enforced but grader-unverified;
+  restored the `auth-notes` DOM XSS smoke (saved markup must render as literal text,
+  not interpreted HTML); and restored the `async-word-counter` enqueue-time
+  processing-state persistence assertion (a job must be persisted as `"processing"`
+  the instant it is enqueued, not only on completion).

scripts/agent-bench/steps/3-build-and-test.sh

@@ -19,6 +19,13 @@ CELL_TMP="/tmp/bench-${TASK:-default}-$$"
 mkdir -p "$CELL_TMP"
 export PW_RESULTS_JSON="${CELL_TMP}/pw-results.json"
 
+# Test-only mock switch, inherited by both the dev server and Playwright. Tasks
+# whose grader needs a server-side test backdoor gate that surface on BLOCKS_MOCK
+# and return null otherwise, so it is inert in a real deployment. Example:
+# cognito-profile's api.getLastCode exposes the most-recently delivered OTP
+# (the grader has no mailbox) only when this is set.
+export BLOCKS_MOCK=true
+
 # Pessimistic defaults up front, updated on success, so a failure still yields well-formed EVIDENCE.
 # build_status defaults "failed" (matches build_succeeded=false); the build step overwrites both.
 {

tasks/README.md

@@ -0,0 +1,43 @@
+# Bench tasks
+
+Each subdirectory is one benchmark task with exactly two files:
+
+- **`PROMPT.md`** — the instructions handed to the building agent.
+- **`test.spec.ts`** — the Playwright grader run against the agent-built app.
+
+The reference app is **not** committed; the agent builds it during the bench,
+then the spec grades the running dev server.
+
+## Intentional per-spec helper duplication
+
+Every spec re-declares the same small helpers inline — `watchErrors(page)`, a
+JSON-RPC `rpc(ctx, method, params)` wrapper, and the `uniq(base)` seed — rather
+than importing them from a shared module. **This duplication is deliberate:**
+each `test.spec.ts` must be a self-contained grader that can be dropped next to
+a single task app and run on its own, with no cross-task imports or build step.
+Keeping the helpers local keeps the specs portable and independently readable;
+do not refactor them into a shared import.
+
+## Harness contract
+
+The bench runs each spec **serially with `workers: 1`** (see
+`scripts/agent-bench/steps/3-build-and-test.sh`). Specs assert against a
+**shared server-side store** (e.g. the presence roster, the file list) and scope
+their assertions to their own `uniq(...)` names rather than to absolute counts.
+Those assertions are race-free **only** under `workers: 1`; running with more
+workers would require reworking them to tolerate concurrent runners. Each spec
+that relies on this carries a `// HARNESS CONTRACT: requires workers:1` banner.
+
+Two independent counters back the helpers so neither perturbs the other:
+
+- **`rpcSeq`** — the JSON-RPC `id` on each request.
+- **`uniqSeq`** — the monotonic component of `uniq(...)` test-data seeds.
+
+## Test-only backdoors (`BLOCKS_MOCK`)
+
+A few tasks need a server-side hook the grader can read because it has no real
+side channel (e.g. `cognito-profile`'s `api.getLastCode`, which surfaces the
+most-recently delivered OTP since the grader has no mailbox). Such hooks MUST be
+gated: annotated `@blocksSkipCodegen` and returning `null` unless
+`process.env.BLOCKS_MOCK === 'true'`, so they are inert in a real deployment.
+The harness exports `BLOCKS_MOCK=true` for the graded dev server.

tasks/async-word-counter/PROMPT.md

@@ -1,41 +1,50 @@
 # Task: Async Word Counter
 
-Build an async word counter in this AWS Blocks app. A user submits some text; the counting happens in a background job. The row for that submission starts as "processing" and flips to "done" with the word count once the job finishes. Results survive a page reload.
+Build an async word counter in this AWS Blocks app. A user submits some text; the counting happens in a **background job**. A submission starts as `"processing"` and flips to `"done"` with its word count once the job finishes. Results are persisted and survive a reload.
+
+The core of this task is **using the framework**: submissions are enqueued and read through an **`api` namespace** whose methods run on the server, do the counting in a **background job block**, and persist each job in a **key/value block** keyed by job id. The page is a thin client that polls that API.
 
 ## Setup (do this first)
 
-The workspace has already been scaffolded. Begin by reading README.md, then do all your edits in this workspace.
+The workspace has already been scaffolded. Begin by reading README.md (and any AGENTS.md), then do all your edits in this workspace.
 
 ## Requirements
 
-1. A user types text into an input and clicks submit.
-2. Submitting enqueues a background job (do the counting in the job — not inline in the request handler) and immediately adds a row for it whose `data-status` is `"processing"`.
-3. The background job counts the words and stores the result in a key/value block keyed by the job id. **Count by whitespace runs only:** trim the text, then split on any run of whitespace (spaces, tabs) — so `one two three four five` is 5, and `"   a   b  "` (leading/trailing + double spaces) is 2. A naive `split(' ')` that counts empty gaps is wrong.
-   - **Punctuation is part of a word, not a separator:** only whitespace separates words, so `hello,world foo.bar-baz!` is **3** words, not 5.
-   - **Unicode and emoji tokens each count as one word:** `café 日本語 🙂 naïve` is **4** words. Count each maximal run of non-whitespace as one word — do **not** use `\w+` / `\W+`, which miscount punctuation and non-ASCII/emoji characters.
-4. **Polling:** the frontend polls for the result; when it's ready the row's `data-status` becomes `"done"` and the row shows the word count.
-5. **Persistence (including in-flight jobs):** persist each submission to the key/value block **when you enqueue it** (status `"processing"`, keyed by job id) — not only when it finishes. On load, **restore the whole list from the store** (every job, including still-`processing` ones) and resume polling — not just the most recent submission. A row reloaded **while still processing** must reappear and still resolve to `"done"` with its count. (An app that tracks the job list only in client memory loses an in-flight row on reload — that fails.)
-6. **Multiple jobs, keyed by job id:** submitting several times enqueues several independent jobs; each gets its own row and its own correct count. Results must be keyed by **job id**, never by the input text — so submitting the **same text twice** produces **two** independent rows, each with its own result that doesn't bleed into the other.
-7. **Input validation:** disable `[data-testid=wc-submit]` whenever the input is empty or whitespace-only (trim before the check); re-enable it once real text is present. Don't enqueue a job for blank input.
+### The `api` namespace (primary surface)
+
+Expose an **`api` namespace** (a `POST /aws-blocks/api` JSON-RPC endpoint) with these methods:
+
+1. **`api.enqueue(text)`** — enqueues a background word-count job for `text` and returns the new job's id as `{ "id": <string> }` (a non-empty job id). It must:
+   - do the counting in a **background job** (the AsyncJob block) — **not** inline in the request handler;
+   - **persist the job at enqueue time** with status `"processing"`, keyed by job id, in the key/value block (not only when it finishes);
+   - **validate the input** — `text` that is empty or whitespace-only (trim first) is rejected with a JSON-RPC **error** envelope and enqueues **no** job.
+2. **`api.getJob(id)`** — returns the job as exactly `{ "id": <string>, "text": <string>, "status": "processing" | "done", "count": <number|null> }`. `count` is the word count once `status === "done"` (and may be `null`/absent while processing). An **unknown** id returns a JSON-RPC **error** envelope (not a bogus success).
+3. **`api.listJobs()`** — returns an array of every job (each the same `{ id, text, status, count }` shape), **restored from the store** — including still-`processing` ones. This is what lets the page rebuild its list on load.
+
+**Word count — count by whitespace runs only.** Trim the text, then split on any run of whitespace (spaces, tabs, newlines): `one two three four five` is `5`, and `"   a   b  "` is `2` (a naive `split(' ')` that counts empty gaps is wrong). **Punctuation is part of a word, not a separator** — `hello,world foo.bar-baz!` is `3`. **Unicode and emoji tokens each count as one word** — `café 日本語 🙂 naïve` is `4`. Count each maximal run of non-whitespace as one word; do **not** use `\w+` / `\W+` (they miscount punctuation and non-ASCII/emoji).
+
+**Keyed by job id, never by text.** Submitting the **same text twice** produces **two** independent jobs, each with its own id and its own result — results must never bleed between jobs or collapse into one.
 
-A single shared list — no login.
+### The page (thin client / light smoke)
+
+1. A text input (`[data-testid=wc-input]`) and a submit button (`[data-testid=wc-submit]`). Disable submit whenever the input is empty or whitespace-only (trim before checking); re-enable it once real text is present.
+2. Submitting calls `api.enqueue`, then the page polls (`api.getJob` / `api.listJobs`) and renders a row per job inside `[data-testid=wc-list]`.
+3. On load, the list is rebuilt from `api.listJobs` (every job, including still-processing ones) and polling resumes — so a row reloaded while still processing reappears and still resolves to `"done"`.
 
 ## Selector contract
 
-The Playwright test grades your work using `data-testid` hooks and one data attribute. Implement them exactly.
+The page smoke test uses these hooks. Implement them exactly.
 
 | Selector | Element | Purpose |
 |---|---|---|
 | `[data-testid=wc-input]` | `<input type="text">` (or `<textarea>`) | Where the user types the text to count |
-| `[data-testid=wc-submit]` | `<button>` | Enqueue an async word-count job for the input text |
+| `[data-testid=wc-submit]` | `<button>` | Enqueue a job (calls `api.enqueue`) |
 | `[data-testid=wc-list]` | container | Wraps every job row |
-| `[data-testid=wc-item]` | one per job, inside the list | The row for a single job; must also render the submitted text as its content |
+| `[data-testid=wc-item]` | one per job, inside the list | The row for a single job; must render the submitted text as its content |
 | `[data-testid=wc-status]` | inside the item | Renders the job's status |
-| `[data-testid=wc-result]` | inside the item | Renders the word count once the job is done |
-
-Set `data-status` on each `[data-testid=wc-item]`: `"processing"` while the job runs, `"done"` once the result is stored. When done, `[data-testid=wc-result]` must show the word count as a bare number (e.g. `5`).
+| `[data-testid=wc-result]` | inside the item | Renders the word count (a bare number, e.g. `5`) once done |
 
-A `[data-testid=wc-item]` must contain the submitted text (the test locates a submission's row via `filter({ hasText: <submitted text> })`, since the job list is shared).
+Set `data-status` on each `[data-testid=wc-item]`: `"processing"` while running, `"done"` once the result is stored. A `[data-testid=wc-item]` must contain the submitted text (the smoke test locates a row via `filter({ hasText: <submitted text> })`).
 
 The mount point for your page is the existing root element. You can replace whatever placeholder content the template ships with.
 
@@ -44,8 +53,8 @@ The mount point for your page is the existing root element. You can replace what
 - Authentication, accounts, per-user lists
 - Cancelling or retrying jobs, progress percentages
 - Counting anything other than whitespace-separated words
-- Styling beyond what makes the test pass
 - Ordering, sorting, filtering, search, pagination
+- Styling beyond what makes the test pass
 
 ## Done means