feat(bench): reshape tasks to api-first framework-focus grading#191
Open
hfurkanbozkurt wants to merge 16 commits into
Open
feat(bench): reshape tasks to api-first framework-focus grading#191hfurkanbozkurt wants to merge 16 commits into
hfurkanbozkurt wants to merge 16 commits into
Conversation
…lign kb-chat prompt
…ng-state smokes, add changeset - #11 dead OTP-gate else: cognito-profile's api.getLastCode test dropped the never-executed `else` (bench always runs BLOCKS_MOCK=true) and documents the production gate is prompt-enforced but grader-unverified; keeps the live mock-mode assertion - #12 restore auth-notes DOM XSS smoke: saved <b>/<i> markup must render as literal text (note-display .locator('b')/('i') toHaveCount 0), alongside the api round-trip equality - #13 restore async-word-counter enqueue-time persistence: a job must be readable as status 'processing' the instant it is enqueued (getJob + listJobs), matching PROMPT lines 3/19/22/32, then resolves to done - #14 fix stale async:129 comment (reuses the request fixture, not a new context) - add empty-frontmatter changeset so the required 'Require changeset' check passes (internal bench/task content only, no packages/* change)
soberm
previously approved these changes
Jul 13, 2026
ahmedhamouda78
previously approved these changes
Jul 13, 2026
…-focus # Conflicts: # scripts/agent-bench/steps/3-build-and-test.sh
…fallback, OTP/AppSetting rubric notes
🦋 Changeset detectedLatest commit: 4dfd91b The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
The post-enqueue read asserted status === 'processing' with a null count, which flakes against a legitimately-fast reference impl whose background job finishes before the read. DONE (20s) is a poll CEILING, not a floor, so accept 'processing' | 'done' for the immediate read (require null count only while 'processing'). The listJobs still-processing surfacing and the final countOf() === 5 assertion are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Reshapes all 10 bench tasks to be API-first ("just use the framework" — Ask E). Core grading now runs through an
apinamespace (JSON-RPC viaPOST /aws-blocks/api) backed by real framework blocks, with the DOM reduced to a light smoke test (mount renders + primary control works). This follows thetasks/observability-apigold-standard pattern: heavyrequest→JSON-RPC assertions, light DOM.Task content only — no harness, scoring, workflow, or model changes. Only
tasks/**(PROMPT.md + test.spec.ts) are touched.Per-task changes (8 reshaped)
auth-notessaveNote/getNotegated by session cookie; overwrite, verbatim/no-HTML, per-user isolation (2 contexts), unauth gatingasync-word-counterenqueue/getJob/listJobs; whitespace/unicode counts, keyed-by-id dedupe, unknown-id error, persistencefile-galleryputFile/listFiles/deleteFile/getDownloadUrl; byte-exact size, real download URL bytes, binary/empty/unicode names, overwritesql-kb-catalogaddProduct/listProducts(id-ordered),searchKb; seeded return/refund FAQoidc-dsql-notesaddNote/listNotesgated by OIDC session; verbatim/SQL-injection-safe, oldest-first ordering, unauth gatingcognito-profilewhoamigated by session; identity, gating, post-signout gating;getLastCodetest hookkb-chat-agentask→{reply,toolsUsed,citations}; real retrieval of seeded facts, deterministic tool output, tool routing, no-fabrication, blank rejectedcollab-presence-boardjoin/listPresent; name-keyed dedupe, verbatim/unicode round-trip, blank rejected, multi-visitor persistenceLeft as-is (already API-first):
observability-api(gold standard),email-digest(API-first hybrid).Flagged — not fully framework-focusable
oidc-dsql-notes,cognito-profile— MODERATE. Auth flows (OIDC redirect, email-OTP) are intrinsically multi-view DOM; core data ops moved toapi, but sign-in stays a DOM smoke. OIDC stub IdP is single-subject, so cross-user isolation isn't demonstrable → tests auth gating instead.collab-presence-board— PARTIALLY-DOM. Realtime fan-out genuinely needs two browser contexts; that ONE check stays DOM. Everything else (roster persistence/dedupe/verbatim/validation) moved toapi.Validation
tsc --noEmit(Playwright types) on all 10 specs → clean.async-word-counter: real reference solution (AsyncJob+KVStore) scaffolded from thebaretemplate, run against the dev server → 12/12 pass (10 api-first core + 2 DOM smoke).tasks/**is not a publishable package →changesets statusandverify-changeset-coverage.tsboth pass with no changeset.Do not merge — opening for review.