diff --git a/.changeset/bench-tasks-framework-focus.md b/.changeset/bench-tasks-framework-focus.md new file mode 100644 index 00000000..654edf76 --- /dev/null +++ b/.changeset/bench-tasks-framework-focus.md @@ -0,0 +1,21 @@ +--- +"@aws-blocks/agent-bench": patch +--- + +test(bench): reshape agent-bench task graders to be framework-surface-first + +Internal CI/bench tooling only — changes are confined to the private, unpublished +`@aws-blocks/agent-bench` workspace and its task content (`tasks/*`). No +published-package (`packages/*`) changes, so this needs no release (empty changeset). + +- The per-task Playwright graders now assert the **framework `api` surface** (the + JSON-RPC `POST /aws-blocks/api` namespace) as the canonical signal, with a thin + page smoke on top, so a task is graded on genuine Building-Block usage rather than + DOM-only behavior that a mock UI could fake. +- Follow-up review hardening (same PR): removed a dead, never-executed OTP-gate + `else` assertion in `cognito-profile` (the bench always runs with `BLOCKS_MOCK=true`) + and documented that the production gate is prompt-enforced but grader-unverified; + restored the `auth-notes` DOM XSS smoke (saved markup must render as literal text, + not interpreted HTML); and restored the `async-word-counter` enqueue-time + processing-state persistence assertion (a job must be persisted as `"processing"` + the instant it is enqueued, not only on completion). diff --git a/scripts/agent-bench/steps/3-build-and-test.sh b/scripts/agent-bench/steps/3-build-and-test.sh index 90f575a8..34b4060f 100755 --- a/scripts/agent-bench/steps/3-build-and-test.sh +++ b/scripts/agent-bench/steps/3-build-and-test.sh @@ -19,6 +19,13 @@ CELL_TMP="/tmp/bench-${TASK:-default}-$$" mkdir -p "$CELL_TMP" export PW_RESULTS_JSON="${CELL_TMP}/pw-results.json" +# Test-only mock switch, inherited by both the dev server and Playwright. Tasks +# whose grader needs a server-side test backdoor gate that surface on BLOCKS_MOCK +# and return null otherwise, so it is inert in a real deployment. Example: +# cognito-profile's api.getLastCode exposes the most-recently delivered OTP +# (the grader has no mailbox) only when this is set. +export BLOCKS_MOCK=true + # Pessimistic defaults up front, updated on success, so a failure still yields well-formed EVIDENCE. # build_status defaults "failed" (matches build_succeeded=false); the build step overwrites both. { diff --git a/tasks/README.md b/tasks/README.md new file mode 100644 index 00000000..0e37da2d --- /dev/null +++ b/tasks/README.md @@ -0,0 +1,43 @@ +# Bench tasks + +Each subdirectory is one benchmark task with exactly two files: + +- **`PROMPT.md`** — the instructions handed to the building agent. +- **`test.spec.ts`** — the Playwright grader run against the agent-built app. + +The reference app is **not** committed; the agent builds it during the bench, +then the spec grades the running dev server. + +## Intentional per-spec helper duplication + +Every spec re-declares the same small helpers inline — `watchErrors(page)`, a +JSON-RPC `rpc(ctx, method, params)` wrapper, and the `uniq(base)` seed — rather +than importing them from a shared module. **This duplication is deliberate:** +each `test.spec.ts` must be a self-contained grader that can be dropped next to +a single task app and run on its own, with no cross-task imports or build step. +Keeping the helpers local keeps the specs portable and independently readable; +do not refactor them into a shared import. + +## Harness contract + +The bench runs each spec **serially with `workers: 1`** (see +`scripts/agent-bench/steps/3-build-and-test.sh`). Specs assert against a +**shared server-side store** (e.g. the presence roster, the file list) and scope +their assertions to their own `uniq(...)` names rather than to absolute counts. +Those assertions are race-free **only** under `workers: 1`; running with more +workers would require reworking them to tolerate concurrent runners. Each spec +that relies on this carries a `// HARNESS CONTRACT: requires workers:1` banner. + +Two independent counters back the helpers so neither perturbs the other: + +- **`rpcSeq`** — the JSON-RPC `id` on each request. +- **`uniqSeq`** — the monotonic component of `uniq(...)` test-data seeds. + +## Test-only backdoors (`BLOCKS_MOCK`) + +A few tasks need a server-side hook the grader can read because it has no real +side channel (e.g. `cognito-profile`'s `api.getLastCode`, which surfaces the +most-recently delivered OTP since the grader has no mailbox). Such hooks MUST be +gated: annotated `@blocksSkipCodegen` and returning `null` unless +`process.env.BLOCKS_MOCK === 'true'`, so they are inert in a real deployment. +The harness exports `BLOCKS_MOCK=true` for the graded dev server. diff --git a/tasks/async-word-counter/PROMPT.md b/tasks/async-word-counter/PROMPT.md index 4b0a5d3d..c6c2d92c 100644 --- a/tasks/async-word-counter/PROMPT.md +++ b/tasks/async-word-counter/PROMPT.md @@ -1,41 +1,50 @@ # Task: Async Word Counter -Build an async word counter in this AWS Blocks app. A user submits some text; the counting happens in a background job. The row for that submission starts as "processing" and flips to "done" with the word count once the job finishes. Results survive a page reload. +Build an async word counter in this AWS Blocks app. A user submits some text; the counting happens in a **background job**. A submission starts as `"processing"` and flips to `"done"` with its word count once the job finishes. Results are persisted and survive a reload. + +The core of this task is **using the framework**: submissions are enqueued and read through an **`api` namespace** whose methods run on the server, do the counting in a **background job block**, and persist each job in a **key/value block** keyed by job id. The page is a thin client that polls that API. ## Setup (do this first) -The workspace has already been scaffolded. Begin by reading README.md, then do all your edits in this workspace. +The workspace has already been scaffolded. Begin by reading README.md (and any AGENTS.md), then do all your edits in this workspace. ## Requirements -1. A user types text into an input and clicks submit. -2. Submitting enqueues a background job (do the counting in the job — not inline in the request handler) and immediately adds a row for it whose `data-status` is `"processing"`. -3. The background job counts the words and stores the result in a key/value block keyed by the job id. **Count by whitespace runs only:** trim the text, then split on any run of whitespace (spaces, tabs) — so `one two three four five` is 5, and `" a b "` (leading/trailing + double spaces) is 2. A naive `split(' ')` that counts empty gaps is wrong. - - **Punctuation is part of a word, not a separator:** only whitespace separates words, so `hello,world foo.bar-baz!` is **3** words, not 5. - - **Unicode and emoji tokens each count as one word:** `café 日本語 🙂 naïve` is **4** words. Count each maximal run of non-whitespace as one word — do **not** use `\w+` / `\W+`, which miscount punctuation and non-ASCII/emoji characters. -4. **Polling:** the frontend polls for the result; when it's ready the row's `data-status` becomes `"done"` and the row shows the word count. -5. **Persistence (including in-flight jobs):** persist each submission to the key/value block **when you enqueue it** (status `"processing"`, keyed by job id) — not only when it finishes. On load, **restore the whole list from the store** (every job, including still-`processing` ones) and resume polling — not just the most recent submission. A row reloaded **while still processing** must reappear and still resolve to `"done"` with its count. (An app that tracks the job list only in client memory loses an in-flight row on reload — that fails.) -6. **Multiple jobs, keyed by job id:** submitting several times enqueues several independent jobs; each gets its own row and its own correct count. Results must be keyed by **job id**, never by the input text — so submitting the **same text twice** produces **two** independent rows, each with its own result that doesn't bleed into the other. -7. **Input validation:** disable `[data-testid=wc-submit]` whenever the input is empty or whitespace-only (trim before the check); re-enable it once real text is present. Don't enqueue a job for blank input. +### The `api` namespace (primary surface) + +Expose an **`api` namespace** (a `POST /aws-blocks/api` JSON-RPC endpoint) with these methods: + +1. **`api.enqueue(text)`** — enqueues a background word-count job for `text` and returns the new job's id as `{ "id": }` (a non-empty job id). It must: + - do the counting in a **background job** (the AsyncJob block) — **not** inline in the request handler; + - **persist the job at enqueue time** with status `"processing"`, keyed by job id, in the key/value block (not only when it finishes); + - **validate the input** — `text` that is empty or whitespace-only (trim first) is rejected with a JSON-RPC **error** envelope and enqueues **no** job. +2. **`api.getJob(id)`** — returns the job as exactly `{ "id": , "text": , "status": "processing" | "done", "count": }`. `count` is the word count once `status === "done"` (and may be `null`/absent while processing). An **unknown** id returns a JSON-RPC **error** envelope (not a bogus success). +3. **`api.listJobs()`** — returns an array of every job (each the same `{ id, text, status, count }` shape), **restored from the store** — including still-`processing` ones. This is what lets the page rebuild its list on load. + +**Word count — count by whitespace runs only.** Trim the text, then split on any run of whitespace (spaces, tabs, newlines): `one two three four five` is `5`, and `" a b "` is `2` (a naive `split(' ')` that counts empty gaps is wrong). **Punctuation is part of a word, not a separator** — `hello,world foo.bar-baz!` is `3`. **Unicode and emoji tokens each count as one word** — `café 日本語 🙂 naïve` is `4`. Count each maximal run of non-whitespace as one word; do **not** use `\w+` / `\W+` (they miscount punctuation and non-ASCII/emoji). + +**Keyed by job id, never by text.** Submitting the **same text twice** produces **two** independent jobs, each with its own id and its own result — results must never bleed between jobs or collapse into one. -A single shared list — no login. +### The page (thin client / light smoke) + +1. A text input (`[data-testid=wc-input]`) and a submit button (`[data-testid=wc-submit]`). Disable submit whenever the input is empty or whitespace-only (trim before checking); re-enable it once real text is present. +2. Submitting calls `api.enqueue`, then the page polls (`api.getJob` / `api.listJobs`) and renders a row per job inside `[data-testid=wc-list]`. +3. On load, the list is rebuilt from `api.listJobs` (every job, including still-processing ones) and polling resumes — so a row reloaded while still processing reappears and still resolves to `"done"`. ## Selector contract -The Playwright test grades your work using `data-testid` hooks and one data attribute. Implement them exactly. +The page smoke test uses these hooks. Implement them exactly. | Selector | Element | Purpose | |---|---|---| | `[data-testid=wc-input]` | `` (or `