You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: project name is CoderCup; codearena- stays only as resource prefix
CodeArena was the working title. Prose now says CoderCup; literal
identifiers (codearena-* AWS resources, CodeArena*Stack stack names,
the Project=CodeArena tag) keep the historical prefix, with a naming
note in design.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/design.md
+20-18Lines changed: 20 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,26 +1,28 @@
1
-
# CodeArena — High-Level Design
1
+
# CoderCup — High-Level Design
2
2
3
3
Status: v0.1 — first articulation of the architecture that the v1 release (codename `codearena-v1`) builds toward.
4
4
Owner: Claude Code (autonomous build, Yunhao Jiao as product owner)
5
5
Last updated: 2026-05-25
6
6
7
-
This document is the cross-cutting architecture reference for CodeArena. It is **not** the project's strategic intent and it is **not** the week-by-week schedule (those live in internal planning docs outside this repo). It is the single source of truth for **how the system is shaped** — components, contracts, data flow, security boundaries.
7
+
This document is the cross-cutting architecture reference for CoderCup. It is **not** the project's strategic intent and it is **not** the week-by-week schedule (those live in internal planning docs outside this repo). It is the single source of truth for **how the system is shaped** — components, contracts, data flow, security boundaries.
8
8
9
-
If you only read one doc to understand CodeArena's architecture, read this one.
9
+
If you only read one doc to understand CoderCup's architecture, read this one.
10
+
11
+
> **Naming note:** CoderCup began under the working title *CodeArena*. AWS resources (`codearena-*` buckets/tables/Lambdas), CDK stack names (`CodeArena*Stack`), and the `Project=CodeArena` tag keep the original prefix — renaming live infrastructure isn't worth the churn. Prose uses the product name, CoderCup.
10
12
11
13
---
12
14
13
15
## 1. Vision and operating principle
14
16
15
-
### 1.1 What CodeArena is
17
+
### 1.1 What CoderCup is
16
18
17
-
CodeArena is **the public leaderboard for AI coding agents**, hosted by TestSprite as the neutral verifier. Each event poses one standardized task; the headline frontier-lab agents (Claude Code, Codex, Anti-Gravity) all ship a deployable solution under identical prompts, time budgets, and environments. TestSprite verifies the deployable, scores it on three dimensions (correctness, bugs caught, cost-efficiency), and publishes everything — raw transcripts, deployed apps, scores — to the public leaderboard.
19
+
CoderCup is **the public leaderboard for AI coding agents**, hosted by TestSprite as the neutral verifier. Each event poses one standardized task; the headline frontier-lab agents (Claude Code, Codex, Anti-Gravity) all ship a deployable solution under identical prompts, time budgets, and environments. TestSprite verifies the deployable, scores it on three dimensions (correctness, bugs caught, cost-efficiency), and publishes everything — raw transcripts, deployed apps, scores — to the public leaderboard.
18
20
19
21
The inaugural event is the **World Cup Code Battle 2026**, with the leaderboard going public on 2026-06-22, three days before the start of the championship's knockout rounds. The task each agent ships is a public web app for predicting championship outcomes. After launch, prediction-accuracy of the deployed apps becomes a live side-metric that updates every 15 minutes during knockout matches.
20
22
21
-
### 1.2 What CodeArena is not
23
+
### 1.2 What CoderCup is not
22
24
23
-
- Not a benchmark for academic AI research. SWE-bench fills that role. CodeArena is positioned for **public-facing** comparison, not laboratory comparison.
25
+
- Not a benchmark for academic AI research. SWE-bench fills that role. CoderCup is positioned for **public-facing** comparison, not laboratory comparison.
24
26
- Not a TestSprite product marketing surface. The TestSprite brand sits in the footer; the leaderboard's job is to be a credible scoreboard, not a funnel. The marketing value accrues to TestSprite by being *the* trusted referee for the AI coding agent comparison.
25
27
- Not a betting site. No monetary stakes, no real-money predictions. Users predict outcomes in the agent-built apps for cosmetic points only.
26
28
- Not a runner-as-a-service for arbitrary user-defined tasks. v1 ships a curated task only. The community-task submission surface (`/submit`) lands in v1, but task selection stays with us.
@@ -37,11 +39,11 @@ Three commitments shape every architectural decision. They are non-negotiable.
37
39
38
40
## 2. System overview
39
41
40
-
CodeArena's runtime decomposes into five components plus an off-system test corpus. The picture below is the steady-state v1 architecture; see §4 for what's deployed today vs scheduled.
42
+
CoderCup's runtime decomposes into five components plus an off-system test corpus. The picture below is the steady-state v1 architecture; see §4 for what's deployed today vs scheduled.
@@ -264,19 +266,19 @@ The two calibration constants (`max_bugs = 20`, `max_usd_imputed = 50`) are plac
264
266
265
267
## 5. Security model
266
268
267
-
CodeArena runs in AWS account **<AWS_ACCOUNT_ID>** in **us-east-1**. The account is **shared with TestSprite production workloads**. Every CodeArena resource enforces a four-rule guardrail; if you are adding a resource to this codebase, it must pass all four:
269
+
CoderCup runs in AWS account **<AWS_ACCOUNT_ID>** in **us-east-1**. The account is **shared with TestSprite production workloads**. Every CoderCup resource enforces a four-rule guardrail; if you are adding a resource to this codebase, it must pass all four:
268
270
269
271
1.**Naming**: prefixed `codearena-`. No exceptions.
270
-
2.**Tagging**: tagged `Project=CodeArena` and `ManagedBy=ClaudeCode` via `Tags.of(app)` in `infra/cdk/bin/codearena.ts`. CDK applies these uniformly; new top-level constructs inherit.
271
-
3.**IAM scoping**: any policy attached to a CodeArena role uses resource ARN patterns matching the `codearena-` prefix. No `Resource: "*"` outside the specific service-level patterns AWS requires (e.g., CloudWatch metric publishing is namespace-conditional, not resource-conditional). The runner role's S3 grants use `Bucket.grantReadWrite(role)` which CDK expands to the bucket ARN and `<bucket-arn>/*` — exactly the codearena buckets, nothing else.
272
-
4.**No cross-system reads**: CodeArena drivers / Lambdas never read TestSprite production DynamoDB tables, never write to TestSprite production S3 buckets, never assume the TestSprite production-tier IAM roles. The TestSprite SaaS that's the referee runs in an isolated environment, so a production-tier outage there cannot affect CodeArena.
272
+
2.**Tagging**: tagged `Project=CoderCup` and `ManagedBy=ClaudeCode` via `Tags.of(app)` in `infra/cdk/bin/codearena.ts`. CDK applies these uniformly; new top-level constructs inherit.
273
+
3.**IAM scoping**: any policy attached to a CoderCup role uses resource ARN patterns matching the `codearena-` prefix. No `Resource: "*"` outside the specific service-level patterns AWS requires (e.g., CloudWatch metric publishing is namespace-conditional, not resource-conditional). The runner role's S3 grants use `Bucket.grantReadWrite(role)` which CDK expands to the bucket ARN and `<bucket-arn>/*` — exactly the codearena buckets, nothing else.
274
+
4.**No cross-system reads**: CoderCup drivers / Lambdas never read TestSprite production DynamoDB tables, never write to TestSprite production S3 buckets, never assume the TestSprite production-tier IAM roles. The TestSprite SaaS that's the referee runs in an isolated environment, so a production-tier outage there cannot affect CoderCup.
273
275
274
-
**Secrets**: there are none managed by CodeArena infrastructure. Each agent CLI uses account-login auth on the runner EC2; those credential files live in `/home/ec2-user/.config/*` on the box, owned by the OS user, never serialized into our IaC or commit history.
276
+
**Secrets**: there are none managed by CoderCup infrastructure. Each agent CLI uses account-login auth on the runner EC2; those credential files live in `/home/ec2-user/.config/*` on the box, owned by the OS user, never serialized into our IaC or commit history.
275
277
276
278
**Public surfaces**:
277
279
-`https://main.d2wicurs2ws3dd.amplifyapp.com` (and eventual custom domain) — public read of leaderboard.
278
280
- CloudFront distribution fronting `codearena-public-data-*` — public read of leaderboard.json, agents/*.json, runs/*/live.jsonl. Bucket itself is BPA-blocked; only the OAC CloudFront principal can read.
279
-
-`https://github.com/TestSprite/CodeArena` — public source, public commit history.
281
+
-`https://github.com/TestSprite/CoderCup` — public source, public commit history.
280
282
281
283
**Private surfaces**:
282
284
-`codearena-runs-*` S3 bucket — private, no public principals.
@@ -289,7 +291,7 @@ CodeArena runs in AWS account **<AWS_ACCOUNT_ID>** in **us-east-1**. The account
289
291
290
292
## 6. Operational model — autonomous build
291
293
292
-
CodeArena is built end-to-end by an autonomous Claude Code session. The build follows three loops:
294
+
CoderCup is built end-to-end by an autonomous Claude Code session. The build follows three loops:
293
295
294
296
1.**Build loop**: design (in `docs/codearena-v1/`) → implement (in `app/`, `runners/`, `scoring/`, `infra/`) → commit → push → Amplify auto-deploys for FE / `cdk deploy` for infra.
295
297
2.**Verify loop**: after a feature lands, invoke the TestSprite verify skill. It runs the relevant TestSprite tests via the CLI, inspects the results, logs friction observations for the TestSprite team. Tests created during the verify loop stay in the project's TestSprite suite — they accumulate into the v1 release suite.
@@ -320,8 +322,8 @@ Two regimes: launch month (May 26 – June 25) and steady state (June 26 onward)
320
322
- EC2 stop-and-start, weekdays-only ≈ ~$15/mo
321
323
- Other infra ≈ $15-25/mo
322
324
323
-
**Out of scope from CodeArena's own AWS budget**:
324
-
- The agent-built apps' hosting (each ships to its own Amplify app post-launch; counted in the agent's `usd_imputed` indirectly via deploy minutes, not CodeArena infra).
325
+
**Out of scope from CoderCup's own AWS budget**:
326
+
- The agent-built apps' hosting (each ships to its own Amplify app post-launch; counted in the agent's `usd_imputed` indirectly via deploy minutes, not CoderCup infra).
325
327
- Inference token spend if we ever bypass the subscriptions (currently NOT planned — all three CLIs are subscription-billed).
326
328
327
329
CloudFront egress is the volatile term during launch week. At $0.085/GB outbound, a 1 TB launch-day surge is ~$85. A separate $500 launch-month allowance is carved out for this overshoot scenario; outside launch week, egress is negligible.
Copy file name to clipboardExpand all lines: infra/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# /infra — AWS CDK stacks for CodeArena
1
+
# /infra — AWS CDK stacks for CoderCup
2
2
3
3
Deploys to **us-east-1**. The AWS account may be shared with other workloads; every resource must be prefixed `codearena-` and tagged `Project=CodeArena ManagedBy=ClaudeCode`. See `/docs/design.md` for the rules.
0 commit comments