Skip to content

Commit 0ee579e

Browse files
study8677study8677
committed
feat: add first review challenge
1 parent c961527 commit 0ee579e

12 files changed

Lines changed: 334 additions & 86 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ AI 时代已经来了，但很多企业仍然在用传统 LeetCode 的方式考
3535
- [评估设计](./docs/zh/evaluation.md)
3636
- [首批题库规划](./docs/zh/challenges.md)
3737
- [V0 执行计划](./plan.md)
38+
- [第一题：SymPy Point2D AI 补丁审查](./challenges/review/001-sympy-point2d-ai-patch/README.zh.md)
3839
- [English](./README_en.md)
3940

4041
## 当前阶段
@@ -49,4 +50,4 @@ V0 先聚焦 20 道高质量题：
4950
## 品牌
5051

5152
- 仓库：`agentcode`
52-
- 域名：`agentcoder.codes`
53+
- 域名：`agentcode.codes`

‎README_en.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ That is what AgentCode is built for: a new practice ground for the skills that m
3535
- [Evaluation Design](./docs/en/evaluation.md)
3636
- [First Challenge Set](./docs/en/challenges.md)
3737
- [V0 Plan](./plan.md)
38+
- [First challenge: SymPy Point2D AI patch review](./challenges/review/001-sympy-point2d-ai-patch/README.en.md)
3839
- [中文](./README.md)
3940

4041
## Current Stage
@@ -49,5 +50,4 @@ It does not start with algorithm Hot100, complex community features, contests, l
4950
## Brand
5051

5152
- Repository: `agentcode`
52-
- Domain: `agentcoder.codes`
53-
53+
- Domain: `agentcode.codes`

‎challenges/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# AgentCode Challenges
2+
3+
This directory stores versioned challenge assets.
4+
5+
Current challenge types:
6+
7+
- `review/`: review an AI-generated or agent-generated patch and decide whether it can be merged.
8+
- `task/`: complete a real engineering task in a starter repository. This will be added later.
9+
10+
Challenge assets should be small, source-traceable, and reproducible. When a challenge is adapted from an upstream open source issue or benchmark, its source must be listed in the challenge README and metadata.

‎challenges/review/001-sympy-point2d-ai-patch/README.en.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Review 001: Can This AI Fix Be Merged?
2+
3+
You are reviewing a patch generated by an AI agent. The patch claims to fix a SymPy `Point2D` bug where `evaluate(False)` can incorrectly raise `Imaginary coordinates are not permitted`.
4+
5+
Your task is not to write code. Your task is to decide whether this PR can be merged. If it cannot be merged, identify the concrete risk, impact, and repair direction.
6+
7+
## Source
8+
9+
This challenge is adapted from a real open source issue:
10+
11+
- Upstream project: SymPy
12+
- Real issue: <https://github.com/sympy/sympy/issues/22684>
13+
- Upstream oracle PR: <https://github.com/sympy/sympy/pull/22714>
14+
- AI incorrect patch source: the CodeStory plausible patch for `sympy-22714`, discussed in *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study*: <https://arxiv.org/abs/2503.15223>
15+
16+
Important: you are reviewing the AI patch in `ai-pr.diff`, not the maintainer patch that SymPy merged upstream.
17+
18+
## Background
19+
20+
SymPy `Point` / `Point2D` objects should reject imaginary coordinates. For example, a coordinate that is clearly `I` should not be accepted.
21+
22+
The real bug was different: under `evaluate(False)`, old logic could raise the following error even when the input did not contain imaginary coordinates:
23+
24+
```text
25+
ValueError: Imaginary coordinates are not permitted.
26+
```
27+
28+
The AI agent generated a patch that tries to bypass this error under `evaluate(False)` and added one regression test.
29+
30+
## What To Review
31+
32+
Read:
33+
34+
- `ai-pr.diff`
35+
36+
Then submit a review:
37+
38+
```text
39+
Can merge? Yes / No
40+
41+
Finding 1:
42+
- Severity:
43+
- Problem:
44+
- Why it matters:
45+
- Suggested fix:
46+
47+
Finding 2:
48+
...
49+
```
50+
51+
## Evaluation Focus
52+
53+
Your answer is evaluated on whether you:
54+
55+
- Correctly decide if the AI PR can be merged.
56+
- Identify whether the patch introduces a regression.
57+
- Explain the semantic boundary between `evaluate(False)`, valid real coordinates, and invalid imaginary coordinates.
58+
- Notice that the AI-added test does not cover the core risk.
59+
- Provide an actionable repair direction instead of only saying "add tests."

‎challenges/review/001-sympy-point2d-ai-patch/README.zh.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Review 001：这个 AI 修复能合并吗？
2+
3+
你正在审核一个 AI agent 生成的补丁。补丁声称修复了 SymPy 中 `Point2D` 在 `evaluate(False)` 下误报 `Imaginary coordinates are not permitted` 的问题。
4+
5+
你的任务不是写代码，而是判断这个 PR 是否可以合并。如果不能合并，需要指出具体风险、影响和建议修复方向。
6+
7+
## 题目来源
8+
9+
本题来自真实开源问题，并改编为 AgentCode Review Mode 题目：
10+
11+
- 上游项目：SymPy
12+
- 真实 issue：<https://github.com/sympy/sympy/issues/22684>
13+
- 上游正确修复 PR：<https://github.com/sympy/sympy/pull/22714>
14+
- AI 错误补丁来源：论文 *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study* 中对 `sympy-22714` 的 CodeStory plausible patch 分析：<https://arxiv.org/abs/2503.15223>
15+
16+
注意：你要审核的是 `ai-pr.diff` 中的 AI 补丁，不是 SymPy 最终合并的正确 PR。
17+
18+
## 背景
19+
20+
SymPy 的 `Point` / `Point2D` 不允许创建带有虚数坐标的点。例如，明确的 `I` 坐标应该被拒绝。
21+
22+
但真实 bug 是：在 `evaluate(False)` 环境下，即使输入并不包含虚数，旧逻辑也可能错误地抛出：
23+
24+
```text
25+
ValueError: Imaginary coordinates are not permitted.
26+
```
27+
28+
AI agent 生成了一个补丁，试图在 `evaluate(False)` 下绕过这个报错，并补了一条测试。
29+
30+
## 你需要审核
31+
32+
请阅读：
33+
34+
- `ai-pr.diff`
35+
36+
然后提交 review 结论：
37+
38+
```text
39+
Can merge? Yes / No
40+
41+
Finding 1:
42+
- Severity:
43+
- Problem:
44+
- Why it matters:
45+
- Suggested fix:
46+
47+
Finding 2:
48+
...
49+
```
50+
51+
## 评分重点
52+
53+
你的答案会重点看：
54+
55+
- 是否能正确判断这个 AI PR 是否可以合并。
56+
- 是否识别它是否引入回归。
57+
- 是否能说明 `evaluate(False)`、合法实数坐标、非法虚数坐标之间的语义边界。
58+
- 是否能指出 AI 新增测试覆盖不足。
59+
- 是否给出可执行的修复建议，而不是只说“加测试”。

‎challenges/review/001-sympy-point2d-ai-patch/ai-pr.diff‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
diff --git a/sympy/geometry/point.py b/sympy/geometry/point.py
2+
index 73c20334d09e..ai0000000000 100644
3+
--- a/sympy/geometry/point.py
4+
+++ b/sympy/geometry/point.py
5+
@@ -149,7 +149,7 @@ def __new__(cls, *args, **kwargs):
6+
if any(coords[dim:]):
7+
raise ValueError('Nonzero coordinates cannot be removed.')
8+
- if any(a.is_number and im(a) for a in coords):
9+
+ if evaluate and any(a.is_number and im(a) for a in coords):
10+
raise ValueError('Imaginary coordinates are not permitted.')
11+
if not all(isinstance(a, Expr) for a in coords):
12+
raise TypeError('Coordinates must be valid SymPy expressions.')
13+
diff --git a/sympy/geometry/tests/test_point.py b/sympy/geometry/tests/test_point.py
14+
index 930d707735e2..ai1111111111 100644
15+
--- a/sympy/geometry/tests/test_point.py
16+
+++ b/sympy/geometry/tests/test_point.py
17+
@@ -1,5 +1,6 @@
18+
from sympy.core.basic import Basic
19+
from sympy.core.numbers import (I, Rational, pi)
20+
+from sympy.core.parameters import evaluate
21+
from sympy.core.singleton import S
22+
from sympy.core.symbol import Symbol
23+
from sympy.core.sympify import sympify
24+
@@ -452,6 +453,12 @@ def test__normalize_dimension():
25+
Point(1, 2, 0), Point(3, 4, 0)]
26+
27+
28+
+def test_issue_22684():
29+
+ # Used to give an error
30+
+ with evaluate(False):
31+
+ Point(1, 2)
32+
+
33+
+
34+
def test_direction_cosine():
35+
p1 = Point3D(0, 0, 0)
36+
p2 = Point3D(1, 1, 1)

‎challenges/review/001-sympy-point2d-ai-patch/expected-findings.json‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"canMerge": false,
3+
"requiredFindings": [
4+
{
5+
"id": "imaginary-coordinate-regression",
6+
"severity": "high",
7+
"summary": "The patch disables the imaginary-coordinate safety check whenever evaluate is false.",
8+
"expectedReasoning": "The original bug is that non-imaginary coordinates can be misclassified under evaluate(False). The fix should not allow clearly imaginary coordinates. With this patch, an input such as Point(I, 2) inside evaluate(False) can bypass the ValueError.",
9+
"acceptableFix": "Do not gate the whole check on evaluate. Keep the safety check active and reject only coordinates whose imaginary part is definitely non-zero, for example by using im(a).is_zero is False."
10+
},
11+
{
12+
"id": "missing-negative-test",
13+
"severity": "medium",
14+
"summary": "The new test covers only the happy path and does not prove that invalid imaginary coordinates are still rejected.",
15+
"expectedReasoning": "A regression test for the original issue is necessary but insufficient. The PR also needs a negative test under evaluate(False), such as asserting that Point(I, 2) still raises ValueError.",
16+
"acceptableFix": "Add tests for both non-imaginary coordinates under evaluate(False) and explicitly imaginary coordinates under evaluate(False)."
17+
}
18+
],
19+
"optionalFindings": [
20+
{
21+
"id": "narrower-oracle-like-predicate",
22+
"severity": "medium",
23+
"summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode.",
24+
"expectedReasoning": "The correctness boundary is about whether a coordinate is definitely imaginary. The global evaluate flag is the wrong abstraction for that decision."
25+
}
26+
],
27+
"disallowedConclusions": [
28+
"The patch can be merged because the added test passes.",
29+
"The correct fix is to skip all imaginary-coordinate checks under evaluate(False)."
30+
]
31+
}

‎challenges/review/001-sympy-point2d-ai-patch/metadata.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"id": "review-001-sympy-point2d-ai-patch",
3+
"mode": "review",
4+
"difficulty": "mid",
5+
"status": "ready",
6+
"title": {
7+
"zh": "这个 AI 修复能合并吗？SymPy Point2D 回归审查",
8+
"en": "Can this AI fix be merged? Reviewing a SymPy Point2D regression"
9+
},
10+
"summary": {
11+
"zh": "审查一个针对真实 SymPy 问题的 AI 补丁：它修好了 evaluate(False) 下的报错，但可能放过了非法的虚数坐标。",
12+
"en": "Review an AI patch for a real SymPy issue: it fixes an evaluate(False) error, but may let invalid imaginary coordinates through."
13+
},
14+
"tags": [
15+
"review",
16+
"swe-bench",
17+
"sympy",
18+
"regression",
19+
"edge-case",
20+
"tests"
21+
],
22+
"source": {
23+
"project": "SymPy",
24+
"upstreamIssue": "https://github.com/sympy/sympy/issues/22684",
25+
"upstreamOraclePullRequest": "https://github.com/sympy/sympy/pull/22714",
26+
"analysisPaper": "https://arxiv.org/abs/2503.15223",
27+
"benchmarkInstance": "sympy-22714"
28+
},
29+
"reviewTarget": {
30+
"file": "ai-pr.diff",
31+
"kind": "adapted-ai-plausible-patch",
32+
"note": "The reviewed patch is adapted from the CodeStory plausible-but-incorrect patch described in the PatchDiff paper. It is not the SymPy maintainer patch that was merged upstream."
33+
}
34+
}

‎challenges/review/001-sympy-point2d-ai-patch/rubric.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Rubric
2+
3+
Total: 100 points.
4+
5+
## Merge Decision: 30
6+
7+
- 30: Says this PR must not be merged.
8+
- 10: Expresses uncertainty but identifies a blocking risk.
9+
- 0: Says it can be merged without qualification.
10+
11+
## Core Regression: 35
12+
13+
- 35: Clearly explains that gating the imaginary-coordinate check behind `evaluate` allows invalid imaginary coordinates when `evaluate(False)` is active.
14+
- 20: Mentions a regression around imaginary coordinates but does not connect it to `evaluate(False)`.
15+
- 10: Vaguely says the condition is too broad.
16+
- 0: Misses the regression.
17+
18+
## Semantic Boundary: 15
19+
20+
- 15: Distinguishes unknown/non-imaginary inputs from clearly imaginary inputs, and explains that only clearly imaginary coordinates should be rejected.
21+
- 8: Mentions edge cases but does not describe the semantic distinction.
22+
- 0: Does not discuss the intended behavior.
23+
24+
## Test Quality: 10
25+
26+
- 10: Notes that the added test only checks the formerly failing happy path and misses a negative test for `Point(I, 2)` or equivalent under `evaluate(False)`.
27+
- 5: Says tests are insufficient without specifying the missing case.
28+
- 0: Treats the test as sufficient.
29+
30+
## Repair Direction: 10
31+
32+
- 10: Suggests keeping the imaginary-coordinate guard active and changing the predicate to a stricter zero check, such as rejecting only when `im(a).is_zero is False`, plus adding regression tests.
33+
- 5: Suggests adding tests but not the implementation direction.
34+
- 0: No actionable fix.

‎docs/en/challenges.md‎

Lines changed: 30 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -39,49 +39,41 @@ The quality bar is more important than breadth. Each challenge should have a con
3939

4040
Challenges should live in the repository as versioned content, not only in the database.
4141

42-
Recommended structure:
42+
Current structure:
4343

4444
```text
45-
content/
46-
challenges/
47-
task/
48-
fix-pagination-boundary/
49-
challenge.yaml
50-
prompt.md
51-
repo/
52-
tests/
53-
public/
54-
hidden/
55-
solution.patch
56-
explanation.md
57-
review/
58-
ai-pr-missing-permission-check/
59-
challenge.yaml
60-
prompt.md
61-
base.diff
62-
ai-pr.diff
63-
rubric.yaml
64-
explanation.md
45+
challenges/
46+
README.md
47+
review/
48+
001-sympy-point2d-ai-patch/
49+
metadata.json
50+
README.zh.md
51+
README.en.md
52+
ai-pr.diff
53+
expected-findings.json
54+
rubric.md
6555
```
6656

67-
`challenge.yaml` defines metadata and execution settings:
68-
69-
```yaml
70-
id: fix-pagination-boundary
71-
mode: task
72-
title: Fix pagination boundary behavior
73-
difficulty: medium
74-
tags:
75-
- backend
76-
- testing
77-
- edge-case
78-
runtime:
79-
image: node:22
80-
install: npm install
81-
test: npm test
82-
limits:
83-
timeoutSeconds: 120
57+
Task Mode challenges will later add `starter/`, `tests/visible/`, `tests/hidden/`, `validator.sh`, and `solution.patch`.
58+
59+
`metadata.json` defines metadata and source attribution:
60+
61+
```json
62+
{
63+
"id": "review-001-sympy-point2d-ai-patch",
64+
"mode": "review",
65+
"difficulty": "mid",
66+
"source": {
67+
"project": "SymPy",
68+
"upstreamIssue": "https://github.com/sympy/sympy/issues/22684",
69+
"upstreamOraclePullRequest": "https://github.com/sympy/sympy/pull/22714",
70+
"analysisPaper": "https://arxiv.org/abs/2503.15223"
71+
}
72+
}
8473
```
8574

8675
This makes exercises reviewable, portable, and reproducible.
8776

77+
## Created Challenges
78+
79+
- [Review 001: Can this AI fix be merged? Reviewing a SymPy Point2D regression](../../challenges/review/001-sympy-point2d-ai-patch/README.en.md)

0 commit comments

Comments
 (0)