You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README_en.md
+6-3Lines changed: 6 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,6 +27,8 @@ That is what AgentCode is built for: a new practice ground for the skills that m
27
27
-**Task Mode**: complete realistic engineering tasks with AI assistance.
28
28
-**Review Mode**: review AI / Agent generated PRs and decide whether they can be merged.
29
29
30
+
The live V0 set currently focuses on Review Mode: 20 traceable AI PR review challenges are available now. Task Mode will become live after the runner, starter repositories, visible tests, and hidden checks are ready.
31
+
30
32
## Docs
31
33
32
34
-[Vision](./docs/en/vision.md)
@@ -36,14 +38,15 @@ That is what AgentCode is built for: a new practice ground for the skills that m
36
38
-[First Challenge Set](./docs/en/challenges.md)
37
39
-[V0 Plan](./plan.md)
38
40
-[First challenge: SymPy Point2D AI patch review](./challenges/review/001-sympy-point2d-ai-patch/README.en.md)
41
+
-[Live challenge list](https://agentcode.codes)
39
42
-[中文](./README.md)
40
43
41
44
## Current Stage
42
45
43
-
V0 focuses on 20 high-quality challenges:
46
+
V0 focuses on 20 high-quality challenges. The current live set is:
44
47
45
-
-10 Task Mode challenges.
46
-
-10 Review Mode challenges.
48
+
-20 Review Mode challenges.
49
+
-Task Mode is planned after the runner and test harness are ready.
47
50
48
51
It does not start with algorithm Hot100, complex community features, contests, leaderboards, or a complete online IDE. The first stage is about challenge quality, evaluation credibility, and a real training loop for AI-era engineering skills.
Copy file name to clipboardExpand all lines: challenges/README.md
+11-2Lines changed: 11 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,16 @@ This directory stores versioned challenge assets.
4
4
5
5
Current challenge types:
6
6
7
-
-`review/`: review an AI-generated or agent-generated patch and decide whether it can be merged.
8
-
-`task/`: complete a real engineering task in a starter repository. This will be added later.
7
+
-`review/`: review an AI-generated or agent-generated patch and decide whether it can be merged. V0 currently ships 20 review challenges.
8
+
-`task/`: complete a real engineering task in a starter repository. This will be added after the runner and test harness are ready.
9
9
10
10
Challenge assets should be small, source-traceable, and reproducible. When a challenge is adapted from an upstream open source issue or benchmark, its source must be listed in the challenge README and metadata.
# Review 001: Can this AI fix be merged? Reviewing a SymPy Point2D regression
2
2
3
-
You are reviewing a patch generated by an AI agent. The patch claims to fix a SymPy `Point2D` bug where `evaluate(False)` can incorrectly raise `Imaginary coordinates are not permitted`.
3
+
Review an AI patch for a real SymPy issue: it fixes an evaluate(False) error but may let invalid imaginary coordinates through.
4
4
5
-
Your task is not to write code. Your task is to decide whether this PR can be merged. If it cannot be merged, identify the concrete risk, impact, and repair direction.
5
+
Your task is to review the adapted AI patch, decide whether it can be merged, and write actionable findings.
6
6
7
-
## Source
7
+
## Sources
8
8
9
-
This challenge is adapted from a real open source issue:
- AI incorrect patch source: the CodeStory plausible patch for `sympy-22714`, discussed in *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study*: <https://arxiv.org/abs/2503.15223>
13
+
The patch in `ai-pr.diff` is an AgentCode adapted plausible-but-incorrect patch for review training, not the upstream maintainer fix.
15
14
16
-
Important: you are reviewing the AI patch in `ai-pr.diff`, not the maintainer patch that SymPy merged upstream.
15
+
## Context
17
16
18
-
## Background
17
+
- Point / Point2D 是 SymPy 的几何点对象,坐标必须是合法 SymPy 表达式,并且不能是明确的虚数坐标。
18
+
- 真实 bug 是 evaluate(False) 环境下普通坐标也可能被旧逻辑误判,从而抛出 Imaginary coordinates are not permitted。
19
+
- AI 补丁把整个虚数坐标检查挂到 evaluate 条件后面,容易把“误拒普通坐标”和“拒绝明确虚数坐标”混为一谈。
19
20
20
-
SymPy `Point` / `Point2D` objects should reject imaginary coordinates. For example, a coordinate that is clearly `I` should not be accepted.
21
-
22
-
The real bug was different: under `evaluate(False)`, old logic could raise the following error even when the input did not contain imaginary coordinates:
23
-
24
-
```text
25
-
ValueError: Imaginary coordinates are not permitted.
26
-
```
27
-
28
-
The AI agent generated a patch that tries to bypass this error under `evaluate(False)` and added one regression test.
29
-
30
-
## What To Review
31
-
32
-
Read:
33
-
34
-
-`ai-pr.diff`
35
-
36
-
Then submit a review:
21
+
## Review Format
37
22
38
23
```text
39
24
Can merge? Yes / No
@@ -44,16 +29,14 @@ Finding 1:
44
29
- Why it matters:
45
30
- Suggested fix:
46
31
47
-
Finding 2:
48
-
...
32
+
Testing:
33
+
- Missing regression or boundary tests:
49
34
```
50
35
51
-
## Evaluation Focus
52
-
53
-
Your answer is evaluated on whether you:
36
+
## Rubric Focus
54
37
55
-
-Correctly decide if the AI PR can be merged.
56
-
-Identify whether the patch introduces a regression.
57
-
-Explain the semantic boundary between `evaluate(False)`, valid real coordinates, and invalid imaginary coordinates.
58
-
-Notice that the AI-added test does not cover the core risk.
59
-
-Provide an actionable repair direction instead of only saying "add tests."
38
+
-Correct merge decision.
39
+
-Core risk: The patch disables the imaginary-coordinate safety check whenever evaluate is false.
Copy file name to clipboardExpand all lines: challenges/review/001-sympy-point2d-ai-patch/expected-findings.json
+7-8Lines changed: 7 additions & 8 deletions
Original file line number
Diff line number
Diff line change
@@ -5,23 +5,22 @@
5
5
"id": "imaginary-coordinate-regression",
6
6
"severity": "high",
7
7
"summary": "The patch disables the imaginary-coordinate safety check whenever evaluate is false.",
8
-
"expectedReasoning": "The original bug is that non-imaginary coordinates can be misclassified under evaluate(False). The fix should not allow clearly imaginary coordinates. With this patch, an input such as Point(I, 2) inside evaluate(False) can bypass the ValueError.",
9
-
"acceptableFix": "Do not gate the whole check on evaluate. Keep the safety check active and reject only coordinates whose imaginary part is definitely non-zero, for example by using im(a).is_zero is False."
8
+
"expectedReasoning": "Point(I, 2) inside evaluate(False) can bypass the ValueError, so the original coordinate contract is weakened.",
9
+
"acceptableFix": "Keep the guard active and reject only coordinates whose imaginary part is definitely non-zero, e.g. im(a).is_zero is False."
10
10
},
11
11
{
12
12
"id": "missing-negative-test",
13
13
"severity": "medium",
14
-
"summary": "The new test covers only the happy path and does not prove that invalid imaginary coordinates are still rejected.",
15
-
"expectedReasoning": "A regression test for the original issue is necessary but insufficient. The PR also needs a negative test under evaluate(False), such as asserting that Point(I, 2) still raises ValueError.",
16
-
"acceptableFix": "Add tests for both non-imaginary coordinates under evaluate(False) and explicitly imaginary coordinates under evaluate(False)."
14
+
"summary": "The new test covers only the happy path.",
15
+
"expectedReasoning": "It proves Point(1, 2) under evaluate(False), but not that invalid imaginary coordinates are still rejected.",
16
+
"acceptableFix": "Add a negative test asserting Point(I, 2) still raises under evaluate(False)."
17
17
}
18
18
],
19
19
"optionalFindings": [
20
20
{
21
-
"id": "narrower-oracle-like-predicate",
21
+
"id": "narrower-predicate",
22
22
"severity": "medium",
23
-
"summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode.",
24
-
"expectedReasoning": "The correctness boundary is about whether a coordinate is definitely imaginary. The global evaluate flag is the wrong abstraction for that decision."
23
+
"summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode."
0 commit comments