Skip to content

Commit 5b807ae

Browse files
committed
feat: add 20 review challenges
1 parent c80639b commit 5b807ae

133 files changed

Lines changed: 6495 additions & 649 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
<p align="center">
88
<a href="./README_en.md">English</a>
99
·
10-
<a href="https://agentcode.codes/challenges/review/001-sympy-point2d-ai-patch">试做第一题</a>
10+
<a href="https://agentcode.codes">试做 20 道 Review 题</a>
1111
·
1212
<a href="./docs/zh/product-direction.md">产品方向</a>
1313
·
@@ -64,17 +64,15 @@ V0 正在构建中,目前仓库已经包含:
6464

6565
- 一个 Next.js + TypeScript 产品原型。
6666
- Review Mode 首页、题目详情页和结构化 review 提交表单。
67-
- 首道真实来源改编题:SymPy `Point2D` AI 补丁审查
67+
- 20 道真实来源改编的 Review Mode 题,覆盖安全、框架、HTTP、数据语义和测试质量
6868
- 可版本化的题库资产结构:题面、metadata、AI diff、expected findings、rubric。
6969
- 评估设计、V0 架构、产品方向和首批题库规划文档。
7070

71-
第一题入口
71+
题库入口
7272

73-
- [在线体验:Review 001](https://agentcode.codes/challenges/review/001-sympy-point2d-ai-patch)
73+
- [在线体验:AgentCode 题库](https://agentcode.codes)
7474
- [Review 001:这个 AI 修复能合并吗?SymPy Point2D 回归审查](./challenges/review/001-sympy-point2d-ai-patch/README.zh.md)
75-
- [题目 metadata](./challenges/review/001-sympy-point2d-ai-patch/metadata.json)
76-
- [AI PR diff](./challenges/review/001-sympy-point2d-ai-patch/ai-pr.diff)
77-
- [评分 rubric](./challenges/review/001-sympy-point2d-ai-patch/rubric.md)
75+
- [首批 20 题清单](./docs/zh/challenges.md)
7876

7977
## 快速开始
8078

@@ -117,6 +115,9 @@ challenges/
117115
ai-pr.diff
118116
expected-findings.json
119117
rubric.md
118+
002-next-middleware-header-bypass/
119+
...
120+
020-sympy-partitions-dict-reuse/
120121
```
121122

122123
这种结构让每道题都能明确回答几个问题:
@@ -128,10 +129,10 @@ challenges/
128129

129130
## V0 路线
130131

131-
AgentCode V0 先聚焦 20 道高质量题,而不是堆功能数量。
132+
AgentCode V0 先聚焦 20 道高质量题,而不是堆功能数量。当前已上线的是 20 道 Review Mode 题;Task Mode 在 runner 接入后再作为可练习题上线。
132133

133-
- 10Task Mode:修 bug、做小 feature、补测试、修并发、修缓存、做兼容性改动
134-
- 10 道 Review Mode:审核 AI PR 是否漏掉权限、边界条件、兼容性、测试质量、性能或可维护性风险
134+
- 20Review Mode:审核 AI PR 是否漏掉权限、边界条件、兼容性、测试质量、性能或可维护性风险
135+
- Task Mode 下一阶段:修 bug、做小 feature、补测试、修并发、修缓存、做兼容性改动
135136
- 评估优先使用确定性检查和结构化 rubric,LLM 只作为辅助反馈层。
136137
- 平台边界保持克制:先做好题目质量、评估可信度和训练闭环。
137138

README_en.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ That is what AgentCode is built for: a new practice ground for the skills that m
2727
- **Task Mode**: complete realistic engineering tasks with AI assistance.
2828
- **Review Mode**: review AI / Agent generated PRs and decide whether they can be merged.
2929

30+
The live V0 set currently focuses on Review Mode: 20 traceable AI PR review challenges are available now. Task Mode will become live after the runner, starter repositories, visible tests, and hidden checks are ready.
31+
3032
## Docs
3133

3234
- [Vision](./docs/en/vision.md)
@@ -36,14 +38,15 @@ That is what AgentCode is built for: a new practice ground for the skills that m
3638
- [First Challenge Set](./docs/en/challenges.md)
3739
- [V0 Plan](./plan.md)
3840
- [First challenge: SymPy Point2D AI patch review](./challenges/review/001-sympy-point2d-ai-patch/README.en.md)
41+
- [Live challenge list](https://agentcode.codes)
3942
- [中文](./README.md)
4043

4144
## Current Stage
4245

43-
V0 focuses on 20 high-quality challenges:
46+
V0 focuses on 20 high-quality challenges. The current live set is:
4447

45-
- 10 Task Mode challenges.
46-
- 10 Review Mode challenges.
48+
- 20 Review Mode challenges.
49+
- Task Mode is planned after the runner and test harness are ready.
4750

4851
It does not start with algorithm Hot100, complex community features, contests, leaderboards, or a complete online IDE. The first stage is about challenge quality, evaluation credibility, and a real training loop for AI-era engineering skills.
4952

challenges/README.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,16 @@ This directory stores versioned challenge assets.
44

55
Current challenge types:
66

7-
- `review/`: review an AI-generated or agent-generated patch and decide whether it can be merged.
8-
- `task/`: complete a real engineering task in a starter repository. This will be added later.
7+
- `review/`: review an AI-generated or agent-generated patch and decide whether it can be merged. V0 currently ships 20 review challenges.
8+
- `task/`: complete a real engineering task in a starter repository. This will be added after the runner and test harness are ready.
99

1010
Challenge assets should be small, source-traceable, and reproducible. When a challenge is adapted from an upstream open source issue or benchmark, its source must be listed in the challenge README and metadata.
11+
12+
Each review challenge contains:
13+
14+
- `metadata.json`
15+
- `README.zh.md`
16+
- `README.en.md`
17+
- `ai-pr.diff`
18+
- `expected-findings.json`
19+
- `rubric.md`
Lines changed: 21 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,24 @@
1-
# Review 001: Can This AI Fix Be Merged?
1+
# Review 001: Can this AI fix be merged? Reviewing a SymPy Point2D regression
22

3-
You are reviewing a patch generated by an AI agent. The patch claims to fix a SymPy `Point2D` bug where `evaluate(False)` can incorrectly raise `Imaginary coordinates are not permitted`.
3+
Review an AI patch for a real SymPy issue: it fixes an evaluate(False) error but may let invalid imaginary coordinates through.
44

5-
Your task is not to write code. Your task is to decide whether this PR can be merged. If it cannot be merged, identify the concrete risk, impact, and repair direction.
5+
Your task is to review the adapted AI patch, decide whether it can be merged, and write actionable findings.
66

7-
## Source
7+
## Sources
88

9-
This challenge is adapted from a real open source issue:
9+
- 原始 Issue: <https://github.com/sympy/sympy/issues/22684>
10+
- 上游正确 PR: <https://github.com/sympy/sympy/pull/22714>
11+
- PatchDiff 论文: <https://arxiv.org/abs/2503.15223>
1012

11-
- Upstream project: SymPy
12-
- Real issue: <https://github.com/sympy/sympy/issues/22684>
13-
- Upstream oracle PR: <https://github.com/sympy/sympy/pull/22714>
14-
- AI incorrect patch source: the CodeStory plausible patch for `sympy-22714`, discussed in *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study*: <https://arxiv.org/abs/2503.15223>
13+
The patch in `ai-pr.diff` is an AgentCode adapted plausible-but-incorrect patch for review training, not the upstream maintainer fix.
1514

16-
Important: you are reviewing the AI patch in `ai-pr.diff`, not the maintainer patch that SymPy merged upstream.
15+
## Context
1716

18-
## Background
17+
- Point / Point2D 是 SymPy 的几何点对象,坐标必须是合法 SymPy 表达式,并且不能是明确的虚数坐标。
18+
- 真实 bug 是 evaluate(False) 环境下普通坐标也可能被旧逻辑误判,从而抛出 Imaginary coordinates are not permitted。
19+
- AI 补丁把整个虚数坐标检查挂到 evaluate 条件后面,容易把“误拒普通坐标”和“拒绝明确虚数坐标”混为一谈。
1920

20-
SymPy `Point` / `Point2D` objects should reject imaginary coordinates. For example, a coordinate that is clearly `I` should not be accepted.
21-
22-
The real bug was different: under `evaluate(False)`, old logic could raise the following error even when the input did not contain imaginary coordinates:
23-
24-
```text
25-
ValueError: Imaginary coordinates are not permitted.
26-
```
27-
28-
The AI agent generated a patch that tries to bypass this error under `evaluate(False)` and added one regression test.
29-
30-
## What To Review
31-
32-
Read:
33-
34-
- `ai-pr.diff`
35-
36-
Then submit a review:
21+
## Review Format
3722

3823
```text
3924
Can merge? Yes / No
@@ -44,16 +29,14 @@ Finding 1:
4429
- Why it matters:
4530
- Suggested fix:
4631
47-
Finding 2:
48-
...
32+
Testing:
33+
- Missing regression or boundary tests:
4934
```
5035

51-
## Evaluation Focus
52-
53-
Your answer is evaluated on whether you:
36+
## Rubric Focus
5437

55-
- Correctly decide if the AI PR can be merged.
56-
- Identify whether the patch introduces a regression.
57-
- Explain the semantic boundary between `evaluate(False)`, valid real coordinates, and invalid imaginary coordinates.
58-
- Notice that the AI-added test does not cover the core risk.
59-
- Provide an actionable repair direction instead of only saying "add tests."
38+
- Correct merge decision.
39+
- Core risk: The patch disables the imaginary-coordinate safety check whenever evaluate is false.
40+
- The intended behavior boundary.
41+
- Missing negative or boundary tests.
42+
- Actionable repair direction.
Lines changed: 16 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,24 @@
1-
# Review 001:这个 AI 修复能合并吗?
1+
# Review 001:这个 AI 修复能合并吗?SymPy Point2D 回归审查
22

3-
你正在审核一个 AI agent 生成的补丁。补丁声称修复了 SymPy 中 `Point2D``evaluate(False)` 下误报 `Imaginary coordinates are not permitted` 的问题。
3+
你正在审核一个 AI agent 生成的补丁。补丁声称修复 SymPy 中 Point2D 在 evaluate(False) 下误报 Imaginary coordinates are not permitted 的问题。
44

5-
你的任务不是写代码,而是判断这个 PR 是否可以合并。如果不能合并,需要指出具体风险、影响和建议修复方向。
5+
你的任务不是写代码,而是判断这个 AI PR 是否可以合并。如果不能合并,需要指出具体风险、影响和建议修复方向。
66

77
## 题目来源
88

9-
本题来自真实开源问题,并改编为 AgentCode Review Mode 题目:
9+
本题来自真实工程问题,并改编为 AgentCode Review Mode 题目:
1010

11-
- 上游项目:SymPy
12-
- 真实 issue:<https://github.com/sympy/sympy/issues/22684>
13-
- 上游正确修复 PR:<https://github.com/sympy/sympy/pull/22714>
14-
- AI 错误补丁来源:论文 *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study* 中对 `sympy-22714` 的 CodeStory plausible patch 分析:<https://arxiv.org/abs/2503.15223>
11+
- 原始 Issue:<https://github.com/sympy/sympy/issues/22684>
12+
- 上游正确 PR:<https://github.com/sympy/sympy/pull/22714>
13+
- PatchDiff 论文:<https://arxiv.org/abs/2503.15223>
1514

16-
注意:你要审核的是 `ai-pr.diff` 中的 AI 补丁,不是 SymPy 最终合并的正确 PR
15+
注意:你要审核的是 `ai-pr.diff` 中的改编 AI 补丁,不是上游最终合并的正确修复
1716

1817
## 背景
1918

20-
SymPy 的 `Point` / `Point2D` 不允许创建带有虚数坐标的点。例如,明确的 `I` 坐标应该被拒绝。
21-
22-
但真实 bug 是:在 `evaluate(False)` 环境下,即使输入并不包含虚数,旧逻辑也可能错误地抛出:
23-
24-
```text
25-
ValueError: Imaginary coordinates are not permitted.
26-
```
27-
28-
AI agent 生成了一个补丁,试图在 `evaluate(False)` 下绕过这个报错,并补了一条测试。
19+
- Point / Point2D 是 SymPy 的几何点对象,坐标必须是合法 SymPy 表达式,并且不能是明确的虚数坐标。
20+
- 真实 bug 是 evaluate(False) 环境下普通坐标也可能被旧逻辑误判,从而抛出 Imaginary coordinates are not permitted。
21+
- AI 补丁把整个虚数坐标检查挂到 evaluate 条件后面,容易把“误拒普通坐标”和“拒绝明确虚数坐标”混为一谈。
2922

3023
## 你需要审核
3124

@@ -44,16 +37,14 @@ Finding 1:
4437
- Why it matters:
4538
- Suggested fix:
4639
47-
Finding 2:
48-
...
40+
Testing:
41+
- Missing regression or boundary tests:
4942
```
5043

5144
## 评分重点
5245

53-
你的答案会重点看:
54-
5546
- 是否能正确判断这个 AI PR 是否可以合并。
56-
- 是否识别它是否引入回归。
57-
- 是否能说明 `evaluate(False)`、合法实数坐标、非法虚数坐标之间的语义边界
58-
- 是否能指出 AI 新增测试覆盖不足
47+
- 是否识别核心风险:The patch disables the imaginary-coordinate safety check whenever evaluate is false.
48+
- 是否说明原有行为边界和 AI patch 改变了什么
49+
- 是否指出新增测试覆盖不足
5950
- 是否给出可执行的修复建议,而不是只说“加测试”。

challenges/review/001-sympy-point2d-ai-patch/ai-pr.diff

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -11,26 +11,12 @@ index 73c20334d09e..ai0000000000 100644
1111
if not all(isinstance(a, Expr) for a in coords):
1212
raise TypeError('Coordinates must be valid SymPy expressions.')
1313
diff --git a/sympy/geometry/tests/test_point.py b/sympy/geometry/tests/test_point.py
14-
index 930d707735e2..ai1111111111 100644
15-
--- a/sympy/geometry/tests/test_point.py
16-
+++ b/sympy/geometry/tests/test_point.py
1714
@@ -1,5 +1,6 @@
1815
from sympy.core.basic import Basic
1916
from sympy.core.numbers import (I, Rational, pi)
2017
+from sympy.core.parameters import evaluate
21-
from sympy.core.singleton import S
22-
from sympy.core.symbol import Symbol
23-
from sympy.core.sympify import sympify
2418
@@ -452,6 +453,12 @@ def test__normalize_dimension():
25-
Point(1, 2, 0), Point(3, 4, 0)]
26-
27-
2819
+def test_issue_22684():
2920
+ # Used to give an error
3021
+ with evaluate(False):
3122
+ Point(1, 2)
32-
+
33-
+
34-
def test_direction_cosine():
35-
p1 = Point3D(0, 0, 0)
36-
p2 = Point3D(1, 1, 1)

challenges/review/001-sympy-point2d-ai-patch/expected-findings.json

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,23 +5,22 @@
55
"id": "imaginary-coordinate-regression",
66
"severity": "high",
77
"summary": "The patch disables the imaginary-coordinate safety check whenever evaluate is false.",
8-
"expectedReasoning": "The original bug is that non-imaginary coordinates can be misclassified under evaluate(False). The fix should not allow clearly imaginary coordinates. With this patch, an input such as Point(I, 2) inside evaluate(False) can bypass the ValueError.",
9-
"acceptableFix": "Do not gate the whole check on evaluate. Keep the safety check active and reject only coordinates whose imaginary part is definitely non-zero, for example by using im(a).is_zero is False."
8+
"expectedReasoning": "Point(I, 2) inside evaluate(False) can bypass the ValueError, so the original coordinate contract is weakened.",
9+
"acceptableFix": "Keep the guard active and reject only coordinates whose imaginary part is definitely non-zero, e.g. im(a).is_zero is False."
1010
},
1111
{
1212
"id": "missing-negative-test",
1313
"severity": "medium",
14-
"summary": "The new test covers only the happy path and does not prove that invalid imaginary coordinates are still rejected.",
15-
"expectedReasoning": "A regression test for the original issue is necessary but insufficient. The PR also needs a negative test under evaluate(False), such as asserting that Point(I, 2) still raises ValueError.",
16-
"acceptableFix": "Add tests for both non-imaginary coordinates under evaluate(False) and explicitly imaginary coordinates under evaluate(False)."
14+
"summary": "The new test covers only the happy path.",
15+
"expectedReasoning": "It proves Point(1, 2) under evaluate(False), but not that invalid imaginary coordinates are still rejected.",
16+
"acceptableFix": "Add a negative test asserting Point(I, 2) still raises under evaluate(False)."
1717
}
1818
],
1919
"optionalFindings": [
2020
{
21-
"id": "narrower-oracle-like-predicate",
21+
"id": "narrower-predicate",
2222
"severity": "medium",
23-
"summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode.",
24-
"expectedReasoning": "The correctness boundary is about whether a coordinate is definitely imaginary. The global evaluate flag is the wrong abstraction for that decision."
23+
"summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode."
2524
}
2625
],
2726
"disallowedConclusions": [

0 commit comments

Comments
 (0)