feat: restructure review challenge flow

Author: study8677

challenges/README.md

@@ -7,13 +7,32 @@ Current challenge types:
 - `review/`: review an AI-generated or agent-generated patch and decide whether it can be merged. V0 currently ships 20 review challenges.
 - `task/`: complete a real engineering task in a starter repository. This will be added after the runner and test harness are ready.
 
-Challenge assets should be small, source-traceable, and reproducible. When a challenge is adapted from an upstream open source issue or benchmark, its source must be listed in the challenge README and metadata.
+Challenge assets should be small, source-traceable, and reproducible. When a challenge is adapted from an upstream open source issue or benchmark, its source must be listed in the challenge metadata.
+
+## Review challenge assets
 
 Each review challenge contains:
 
-- `metadata.json`
-- `README.zh.md`
-- `README.en.md`
-- `ai-pr.diff`
-- `expected-findings.json`
-- `rubric.md`
+- `metadata.json` — challenge definition. Pre-submit content lives in `title` / `summary` / `tags` / `pr` / `background`; post-submit content lives in `analysis` / `behaviorChecks` / `learningGoal` / `source.references`.
+- `ai-pr.diff` — the patch under review, presented as a confident AI-authored PR.
+- `src-*.py` / `src-*.ts` / ... — pre-patch source excerpt(s): the minimal sufficient context.
+- `expected-findings.json` — answer key: `canMerge`, required/optional findings with `matchTerms` for local scoring. Spoiler file, hidden from the challenge file browser.
+- `rubric.md` — structured scoring rubric. Spoiler file.
+- `README.zh.md` / `README.en.md` — GitHub-facing brief. Mirrors the pre-submit surface; answers are only pointed to, never inlined.
+
+## The answer distribution is part of the design
+
+Not every challenge is a trap. Some patches are adapted from the correct upstream fix and the right call is to approve them. The pre-submit surface of a mergeable challenge must be indistinguishable from a trap challenge — reviewers earn points by calibrated judgment, not by always answering "request changes".
+
+## Authoring self-check (every challenge must pass all of these)
+
+1. **No spoilers before submit.** Read the full pre-submit surface (title, summary, tags, PR description, background, README, comments inside the diff). If a competent reader can infer the flaw — or infer that there is no flaw — from that surface alone, the challenge fails. The PR speaks only in the author's confident voice.
+2. **Minimal sufficient context.** Every required finding must be derivable from the challenge files alone (diff + source excerpt + neutral background). No reliance on knowledge outside the platform. If discovering the flaw requires seeing code we did not include, include it.
+3. **A senior engineer must have to think.** Looking only at the PR (not at any hints), reaching the right conclusion should take genuine analysis, not pattern-matching a famous CVE from the library name.
+4. **The answer cannot be copied from the prompt.** No pre-submit text may double as an acceptable answer.
+5. **Tests inside the diff are part of the trap or part of the evidence.** Trap patches carry tests that look relevant but do not cover the risk; correct patches carry tests a reviewer can legitimately trust. Say which in `analysis`.
+6. **Upstream traceability.** `source` must link the real issue/PR/advisory, and `reviewTarget.kind` must state whether the diff is an adapted plausible-but-incorrect patch or an adapted correct fix.
+
+## Using AI while solving is allowed
+
+The platform assumes reviewers may use any AI tool — the trained skill is owning the judgment: giving evidence (line numbers, counterexample inputs, impact chains) and being accountable for the merge decision. Challenges must therefore reward specific evidence rather than generic vulnerability vocabulary, and reference answers are revealed only after a submission.

challenges/review/001-sympy-point2d-ai-patch/README.en.md

@@ -1,27 +1,28 @@
-# Review 001: Can this AI fix be merged? Reviewing a SymPy Point2D regression
+# Review 001: SymPy Point: fix spurious coordinate rejection under evaluate(False)
 
-Review an AI patch for a real SymPy issue: it fixes an evaluate(False) error but may let invalid imaginary coordinates through.
+You are a reviewer on this repository. An AI agent submitted the PR below. CI is green and the decision is yours.
 
-Your task is to review the adapted AI patch, decide whether it can be merged, and write actionable findings.
+## PR description (from the author)
 
-## Sources
+> **Fix Point raising 'Imaginary coordinates are not permitted' with evaluate(False)**
+>
+> Parsing Point2D(Integer(1), Integer(2)) inside `with evaluate(False)` raises 'Imaginary coordinates are not permitted' (issue #22684): with automatic evaluation off, im(1) stays as an unevaluated im(1) expression instead of simplifying to 0, so the truthiness test in the coordinate validation treats perfectly real coordinates as having a non-zero imaginary part.
+>
+> This PR runs the imaginary-coordinate validation only when evaluation is enabled — the check relies on im() being evaluated, so it cannot give a reliable verdict while evaluation is off.
+>
+> Added test_issue_22684 reproducing the original failure and verifying Point(1, 2) constructs fine under evaluate(False). All existing tests pass.
 
-- 原始 Issue: <https://github.com/sympy/sympy/issues/22684>
-- 上游正确 PR: <https://github.com/sympy/sympy/pull/22714>
-- PatchDiff 论文: <https://arxiv.org/abs/2503.15223>
+## What to review
 
-The patch in `ai-pr.diff` is an AgentCode adapted plausible-but-incorrect patch for review training, not the upstream maintainer fix.
+Read:
 
-## Context
+- `ai-pr.diff` — the patch under review
+- `src-point.py` — pre-patch excerpt of Point.__new__ (minimal sufficient context)
 
-- Point / Point2D 是 SymPy 的几何点对象，坐标必须是合法 SymPy 表达式，并且不能是明确的虚数坐标。
-- 真实 bug 是 evaluate(False) 环境下普通坐标也可能被旧逻辑误判，从而抛出 Imaginary coordinates are not permitted。
-- AI 补丁把整个虚数坐标检查挂到 evaluate 条件后面，容易把“误拒普通坐标”和“拒绝明确虚数坐标”混为一谈。
-
-## Review Format
+Then submit your review:
 
 ```text
-Can merge? Yes / No
+Can merge? Yes / No / Need more info
 
 Finding 1:
 - Severity:
@@ -30,13 +31,18 @@ Finding 1:
 - Suggested fix:
 
 Testing:
-- Missing regression or boundary tests:
+- What does the new test actually prove? What is missing?
 ```
 
-## Rubric Focus
+## Background
+
+- Point / Point2D are the geometric point objects in SymPy. The constructor validates its inputs; one documented contract is that explicitly imaginary coordinates (e.g. Point(I, 2)) are rejected with a ValueError.
+- evaluate(False) globally disables automatic evaluation: calls like im(1) stay in unevaluated form instead of simplifying to 0. SymPy property queries (such as is_zero) use three-valued logic: True / False / None (unknown).
+
+## Answers and analysis
+
+Reference answers live in `expected-findings.json` and `rubric.md` (spoilers). On the website they are revealed after you submit a review.
+
+## Source
 
-- Correct merge decision.
-- Core risk: The patch disables the imaginary-coordinate safety check whenever evaluate is false.
-- The intended behavior boundary.
-- Missing negative or boundary tests.
-- Actionable repair direction.
+Adapted from a real engineering issue (the `ai-pr.diff` you review is an AgentCode training adaptation, not the upstream fix). Upstream links are in the `source` field of `metadata.json`; read them after attempting the challenge.

challenges/review/001-sympy-point2d-ai-patch/README.zh.md

@@ -1,35 +1,28 @@
-# Review 001：这个 AI 修复能合并吗？SymPy Point2D 回归审查
+# Review 001：SymPy Point：修复 evaluate(False) 下的坐标校验误报
 
-你正在审核一个 AI agent 生成的补丁。补丁声称修复 SymPy 中 Point2D 在 evaluate(False) 下误报 Imaginary coordinates are not permitted 的问题。
+你是这个仓库的 reviewer。一个 AI agent 提交了下面这个 PR，CI 全绿，等待你的结论。
 
-你的任务不是写代码，而是判断这个 AI PR 是否可以合并。如果不能合并，需要指出具体风险、影响和建议修复方向。
+## PR 描述（作者提供）
 
-## 题目来源
-
-本题来自真实工程问题，并改编为 AgentCode Review Mode 题目：
-
-- 原始 Issue：<https://github.com/sympy/sympy/issues/22684>
-- 上游正确 PR：<https://github.com/sympy/sympy/pull/22714>
-- PatchDiff 论文：<https://arxiv.org/abs/2503.15223>
-
-注意：你要审核的是 `ai-pr.diff` 中的改编 AI 补丁，不是上游最终合并的正确修复。
-
-## 背景
-
-- Point / Point2D 是 SymPy 的几何点对象，坐标必须是合法 SymPy 表达式，并且不能是明确的虚数坐标。
-- 真实 bug 是 evaluate(False) 环境下普通坐标也可能被旧逻辑误判，从而抛出 Imaginary coordinates are not permitted。
-- AI 补丁把整个虚数坐标检查挂到 evaluate 条件后面，容易把“误拒普通坐标”和“拒绝明确虚数坐标”混为一谈。
+> **Fix Point raising 'Imaginary coordinates are not permitted' with evaluate(False)**
+>
+> 在 with evaluate(False) 下解析 Point2D(Integer(1), Integer(2)) 会抛出 Imaginary coordinates are not permitted（issue #22684）：关闭自动求值后 im(1) 不会化简成 0，而是保持未求值的 im(1) 表达式，在坐标校验的真值判断里被当成了非零虚部，普通实数坐标因此根本无法构造。
+>
+> 这个 PR 让虚数坐标校验只在 evaluate 开启时执行——这条校验依赖 im() 的求值结果，在求值被关闭时它给不出可靠结论。
+>
+> 新增 test_issue_22684 复现原始报错场景，验证 evaluate(False) 下 Point(1, 2) 能正常创建。现有测试全部通过。
 
 ## 你需要审核
 
 请阅读：
 
-- `ai-pr.diff`
+- `ai-pr.diff` — 待审核的补丁
+- `src-point.py` — 补丁前 Point.__new__ 的源码节选（最小充分上下文）
 
 然后提交 review 结论：
 
 ```text
-Can merge? Yes / No
+Can merge? Yes / No / Need more info
 
 Finding 1:
 - Severity:
@@ -38,13 +31,18 @@ Finding 1:
 - Suggested fix:
 
 Testing:
-- Missing regression or boundary tests:
+- 新增测试证明了什么？还缺什么？
 ```
 
-## 评分重点
+## 背景
+
+- Point / Point2D 是 SymPy 几何模块的点对象。构造函数会对坐标做一系列入参校验，其中一条契约是：明确为虚数的坐标（如 Point(I, 2)）会被 ValueError 拒绝。
+- evaluate(False) 会全局关闭自动求值：像 im(1) 这样的函数调用会保持未求值形式，而不是立即化简为 0。SymPy 的性质查询（如 is_zero）采用三值逻辑：True / False / None（未知）。
+
+## 答案与解析
+
+参考答案在 `expected-findings.json` 和 `rubric.md` 中（剧透注意）。在线做题时提交 review 后自动展示。
+
+## 题目来源
 
-- 是否能正确判断这个 AI PR 是否可以合并。
-- 是否识别核心风险：The patch disables the imaginary-coordinate safety check whenever evaluate is false.
-- 是否说明原有行为边界和 AI patch 改变了什么。
-- 是否指出新增测试覆盖不足。
-- 是否给出可执行的修复建议，而不是只说“加测试”。
+本题改编自真实工程问题（你审核的 `ai-pr.diff` 是 AgentCode 改编的训练补丁，不是上游最终修复）。上游链接在 `metadata.json` 的 `source` 字段中，建议做完题再看。

challenges/review/001-sympy-point2d-ai-patch/ai-pr.diff

@@ -1,22 +1,43 @@
 diff --git a/sympy/geometry/point.py b/sympy/geometry/point.py
-index 73c20334d09e..ai0000000000 100644
+index 73c20334d..5c8636ee6 100644
 --- a/sympy/geometry/point.py
 +++ b/sympy/geometry/point.py
-@@ -149,7 +149,7 @@ def __new__(cls, *args, **kwargs):
+@@ -149,9 +149,12 @@ def __new__(cls, *args, **kwargs):
          if any(coords[dim:]):
              raise ValueError('Nonzero coordinates cannot be removed.')
 -        if any(a.is_number and im(a) for a in coords):
++        # With automatic evaluation disabled, im(a) stays unevaluated (e.g.
++        # im(1)), which made perfectly real coordinates look imaginary
++        # (issue #22684). Only run this validation when evaluation is on.
 +        if evaluate and any(a.is_number and im(a) for a in coords):
              raise ValueError('Imaginary coordinates are not permitted.')
          if not all(isinstance(a, Expr) for a in coords):
              raise TypeError('Coordinates must be valid SymPy expressions.')
+
+         # pad with zeros appropriately
+         coords = coords[:dim] + (S.Zero,)*(dim - len(coords))
 diff --git a/sympy/geometry/tests/test_point.py b/sympy/geometry/tests/test_point.py
+index 930d7fec2..8a1c2b3d4 100644
+--- a/sympy/geometry/tests/test_point.py
++++ b/sympy/geometry/tests/test_point.py
 @@ -1,5 +1,6 @@
  from sympy.core.basic import Basic
- from sympy.core.numbers import (I, Rational, pi)
+ from sympy.core.numbers import (I, Float, Rational, pi)
 +from sympy.core.parameters import evaluate
-@@ -452,6 +453,12 @@ def test__normalize_dimension():
+ from sympy.core.singleton import S
+ from sympy.core.symbol import Symbol
+ from sympy.core.sympify import sympify
+@@ -452,6 +453,13 @@ def test__normalize_dimension():
+     assert Point._normalize_dimension(Point(1, 2), Point(3, 4, 0)) == [
+         Point(1, 2, 0), Point(3, 4, 0)]
+
++
 +def test_issue_22684():
-+    # Used to give an error
++    # Used to raise 'Imaginary coordinates are not permitted.'
 +    with evaluate(False):
-+        Point(1, 2)
++        p = Point(1, 2)
++    assert p == Point(1, 2)
++
+
+ def test_direction_cosine():
+     p1 = Point3D(0, 0, 0)

challenges/review/001-sympy-point2d-ai-patch/expected-findings.json

@@ -1,26 +1,53 @@
 {
   "canMerge": false,
+  "mergeRationale": "Gating the imaginary-coordinate check on evaluate disables a documented validation: Point(I, 2) is silently accepted under evaluate(False).",
   "requiredFindings": [
     {
       "id": "imaginary-coordinate-regression",
       "severity": "high",
       "summary": "The patch disables the imaginary-coordinate safety check whenever evaluate is false.",
       "expectedReasoning": "Point(I, 2) inside evaluate(False) can bypass the ValueError, so the original coordinate contract is weakened.",
-      "acceptableFix": "Keep the guard active and reject only coordinates whose imaginary part is definitely non-zero, e.g. im(a).is_zero is False."
+      "acceptableFix": "Keep the guard active and reject only coordinates whose imaginary part is definitely non-zero, e.g. im(a).is_zero is False.",
+      "matchTerms": [
+        ["point(i"],
+        ["虚数", "evaluate"],
+        ["虚数", "跳过"],
+        ["虚数", "绕过"],
+        ["虚数", "不再"],
+        ["imaginary", "skip"],
+        ["imaginary", "bypass"],
+        ["imaginary", "disable"]
+      ]
     },
     {
       "id": "missing-negative-test",
       "severity": "medium",
       "summary": "The new test covers only the happy path.",
       "expectedReasoning": "It proves Point(1, 2) under evaluate(False), but not that invalid imaginary coordinates are still rejected.",
-      "acceptableFix": "Add a negative test asserting Point(I, 2) still raises under evaluate(False)."
+      "acceptableFix": "Add a negative test asserting Point(I, 2) still raises under evaluate(False).",
+      "matchTerms": [
+        ["测试", "虚数"],
+        ["测试", "point(i"],
+        ["test", "imaginary"],
+        ["test", "point(i"],
+        ["负向", "测试"],
+        ["negative", "test"],
+        ["happy path"]
+      ]
     }
   ],
   "optionalFindings": [
     {
       "id": "narrower-predicate",
       "severity": "medium",
-      "summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode."
+      "summary": "The predicate should encode certainty about the imaginary part, not the global evaluation mode.",
+      "matchTerms": [
+        ["is_zero"],
+        ["确定", "非零"],
+        ["definitely"],
+        ["三值"],
+        ["unknown", "reject"]
+      ]
     }
   ],
   "disallowedConclusions": [