|
1 | | -# Perturbation Benchmark (v2) |
| 1 | +# Perturbation Benchmark |
2 | 2 |
|
3 | | -Benchmark for evaluating how well LLM reviewers detect **seeded errors** in mathematical papers. The pipeline takes real arxiv papers, injects controlled perturbations, runs automated reviews, and measures recall. |
| 3 | +Do AI review systems catch errors deliberately injected into otherwise-good |
| 4 | +papers? This benchmark takes real arXiv papers, injects controlled errors, runs |
| 5 | +automated reviews, and measures recall (the fraction of injected errors a system |
| 6 | +detects). |
| 7 | + |
| 8 | +The paper has the full method and results |
| 9 | +([arXiv:2606.19749](https://arxiv.org/abs/2606.19749)). This README covers how to |
| 10 | +reproduce a run. |
4 | 11 |
|
5 | 12 | ## Pipeline |
6 | 13 |
|
7 | 14 | ``` |
8 | | -extract → generate → validate → inject → review → score |
| 15 | +extract → generate → validate → verify → inject → review → score |
9 | 16 | ``` |
10 | 17 |
|
11 | | -1. **Extract** (`extract.py`): Identify candidate math spans (equations, symbols) in the paper. |
12 | | -2. **Generate** (`generate.py`): Use an LLM to create perturbations for each candidate span. |
13 | | -3. **Validate** (`validate.py`): Check that perturbations are valid (original exists, no overlaps, no garbled text). |
14 | | -4. **Inject** (`inject.py`): Apply valid perturbations to produce a corrupted paper. |
15 | | -5. **Review**: Run `openaireview review` on the corrupted paper with each (model, method) combination. |
16 | | -6. **Score** (`score.py`): Compare review comments against the perturbation manifest using fuzzy substring matching + LLM-as-judge. |
| 18 | +- **extract → generate → validate → verify → inject** (the `openaireview perturb` |
| 19 | + CLI, driven by `perturb_automated.py`): find candidate spans in a paper, |
| 20 | + generate candidate errors, validate and verify them, and inject the kept ones |
| 21 | + to produce a corrupted paper plus a ground-truth manifest. |
| 22 | +- **review** (`run_benchmark.py`): run each (model, method) over the corrupted paper. |
| 23 | +- **score** (`run_benchmark.py`): match each review comment against the manifest |
| 24 | + and compute recall. |
17 | 25 |
|
18 | | -## Error Types |
| 26 | +## Error types |
19 | 27 |
|
20 | | -### Surface errors |
21 | | -Minimal single-token edits to math expressions: |
22 | | -- `operator_or_sign` — flip `+`/`-`, `≤`/`≥`, `∪`/`∩` |
23 | | -- `symbol_binding` — swap a symbol (`α`→`β`) |
24 | | -- `index_or_subscript` — change sub/superscript (`x_i`→`x_{i+1}`) |
25 | | -- `numeric_parameter` — change a number (`0.5`→`0.25`) |
| 28 | +Injected errors span four categories (the underlying error type is in parentheses): |
26 | 29 |
|
27 | | -### Formal errors |
28 | | -Deeper structural corruptions to definitions, theorems, and proofs: |
29 | | -- `def_wrong`, `thm_wrong_condition`, `thm_wrong_conclusion`, `thm_wrong_scope` |
30 | | -- `proof_wrong_direction`, `proof_missing_case`, `proof_wrong_assumption`, `proof_mismatch` |
| 30 | +- **Surface** (`surface`): minimal single-token math edits (flip an operator or sign, change a number, alter a subscript). |
| 31 | +- **Claim** (`statement_empirical`, `claim_theoretical`): a stated empirical or theoretical claim is made wrong. |
| 32 | +- **Reasoning** (`logic`): a logical or derivation step is broken. |
| 33 | +- **Experimental** (`experimental`): an experimental-design or setup detail is corrupted. |
31 | 34 |
|
32 | | -## Quick Start |
| 35 | +## Quick start |
33 | 36 |
|
34 | | -```bash |
35 | | -# Install benchmark dependencies |
36 | | -pip install -e ".[benchmarks]" |
| 37 | +Install the benchmark dependencies and set an API key: |
37 | 38 |
|
38 | | -# Run a single config (prepare → review → score → report) |
39 | | -python benchmarks/perturbation/run_benchmark.py benchmarks/perturbation/configs/default.yaml |
| 39 | +```bash |
| 40 | +uv pip install -e ".[benchmarks]" |
| 41 | +export OPENROUTER_API_KEY=... |
| 42 | +cd benchmarks/perturbation |
| 43 | +``` |
40 | 44 |
|
41 | | -# Run only specific stages |
42 | | -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages prepare,review |
43 | | -python benchmarks/perturbation/run_benchmark.py configs/default.yaml --stages score |
| 45 | +Run the pipeline one stage at a time: |
44 | 46 |
|
45 | | -# Run many domains in one process (workers stay busy across config boundaries) |
46 | | -python benchmarks/perturbation/run_benchmark.py --configs configs/full_*.yaml \ |
47 | | - --parallel-openaireview 2 --parallel-coarse 8 |
| 47 | +```bash |
| 48 | +# 1. Sample papers from arXiv and generate perturbations (extract→...→inject). |
| 49 | +# Writes corrupted papers + manifests under results/perturbations/<category>/<error_type>/. |
| 50 | +python perturb_automated.py --arxiv-category "math.*" --category theoretical \ |
| 51 | + --error-type all --target 10 --min-year 2015 |
| 52 | + |
| 53 | +# Point the config's input_dir at step 1's output, then run the benchmark stages: |
| 54 | +python run_benchmark.py configs/default.yaml --stages prepare # 2. stage corrupted papers |
| 55 | +python run_benchmark.py configs/default.yaml --stages review # 3. run the reviews |
| 56 | +python run_benchmark.py configs/default.yaml --stages score # 4. score against the manifests |
| 57 | +python run_benchmark.py configs/default.yaml --stages report # 5. aggregate the recall tables |
48 | 58 | ``` |
49 | 59 |
|
50 | | -`run_benchmark.py` selects a review system per config via the `system:` field |
51 | | -(`openaireview` | `coarse` | `reviewer3`); see `systems/README.md` for setup |
52 | | -of the third-party systems. |
53 | | - |
54 | | -## Configuration |
| 60 | +Run several stages at once with `--stages prepare,review,score,report`, or sweep |
| 61 | +many configs with `--configs configs/*.yaml`. `run_benchmark.py` selects the |
| 62 | +review system per config via the `system:` field (`openaireview` | `coarse` | |
| 63 | +`reviewer3`). See `systems/README.md` for setting up the external systems. |
55 | 64 |
|
56 | | -Configs are YAML files in `configs/`. Copy `default.yaml` and edit to create experiment variants. Committed configs serve as the experiment log. |
| 65 | +## Config schema |
57 | 66 |
|
58 | 67 | ```yaml |
59 | | -max_papers: 5 |
60 | | -length: short # short (2k-7k words) | medium (7k-17k) | long (>17k) |
61 | | -error_type: surface # surface | formal | all |
62 | | -score_method: llm # llm | fuzzy | semantic |
63 | | - |
64 | | -perturb_model: google/gemini-3-flash-preview |
65 | | -score_model: google/gemini-3-flash-preview |
66 | | - |
67 | | -review_models: |
68 | | - - google/gemini-3-flash-preview |
69 | | - - z-ai/glm-4.6 |
70 | | - |
71 | | -review_methods: |
| 68 | +system: openaireview # openaireview | coarse | reviewer3 |
| 69 | +input_dir: results/perturbations/math_all # corrupted papers from perturb_automated.py |
| 70 | +results_dir: results/default # staged papers, reviews, scores |
| 71 | + |
| 72 | +models: # backbone models to review with |
| 73 | + - openai/gpt-5.5 |
| 74 | + - deepseek/deepseek-v4-flash |
| 75 | +methods: # zero_shot | local | progressive |
72 | 76 | - zero_shot |
73 | 77 | - progressive |
74 | 78 |
|
75 | | -results_dir: benchmarks/perturbation/results |
| 79 | +score_method: llm # llm | fuzzy | semantic |
| 80 | +score_model: google/gemini-3-flash-preview # the llm judge |
76 | 81 | ``` |
77 | 82 |
|
78 | | -Papers are streamed from the [proof-pile](https://huggingface.co/datasets/hoskinson-center/proof-pile) dataset and binned by word count. |
| 83 | +Configs in `configs/` are local except the committed `default.yaml`. Copy it to |
| 84 | +define experiment variants. |
79 | 85 |
|
80 | | -## Results Layout |
81 | | -
|
82 | | -``` |
83 | | -<results_dir>/ |
84 | | - config.yaml # resolved config |
85 | | - perturb/<error_type>/paper_001/ |
86 | | - paper_001.md # original paper |
87 | | - paper-001_clean.md # clean copy |
88 | | - paper-001_corrupted.md # with injected errors |
89 | | - paper-001_perturbations.json # ground-truth manifest |
90 | | - <model_slug>/<error_type>/<method>/paper_001/ |
91 | | - review/*.json # review results |
92 | | - score/<score_method>/*_score.json # recall scores |
93 | | -``` |
| 86 | +## Scoring |
94 | 87 |
|
95 | | -## Reports |
| 88 | +A comment counts as detecting an injected error when it passes two stages: |
96 | 89 |
|
97 | | -Experiment reports are in `reports/`. See `reports/surface_3models_short_medium.md` for the first benchmark run (3 cheap models, 10 papers, surface errors). |
| 90 | +1. **Substring match** — the perturbed text approximately covers the comment's |
| 91 | + quote (normalized, 0.75 coverage), so the judge only sees comments pointing at |
| 92 | + the right span. |
| 93 | +2. **LLM judge** — Gemini-3 Flash Preview rates whether the comment's explanation |
| 94 | + identifies the same error as the manifest's `why_wrong`, on a 1-5 scale, and a |
| 95 | + rating >= 3 counts as detected. |
98 | 96 |
|
99 | | -### Generating report statistics |
| 97 | +Recall is the fraction of injected errors detected, reported per (model, method) |
| 98 | +and per error category by the `report` stage. |
100 | 99 |
|
101 | | -After a pipeline run completes, use `generate_report.py` to aggregate results across all papers, models, and methods: |
| 100 | +## Results layout |
102 | 101 |
|
103 | | -```bash |
104 | | -python benchmarks/perturbation/generate_report.py benchmarks/perturbation/results_short |
| 102 | +``` |
| 103 | +<results_dir>/ |
| 104 | + perturb/<error_type>/paper_NNN/ |
| 105 | + paper_NNN_corrupted.md # injected paper |
| 106 | + paper_NNN_perturbations.json # ground-truth manifest |
| 107 | + <model>/<error_type>/<method>/paper_NNN/ |
| 108 | + review/*.json # review output |
| 109 | + score/<score_method>/*_score.json # recall scores |
105 | 110 | ``` |
106 | 111 |
|
107 | | -This prints markdown tables to stdout covering: |
108 | | -- Configuration summary |
109 | | -- Ground truth counts by length and error type |
110 | | -- Recall by model × method (split by paper length and overall) |
111 | | -- Recall by error type |
112 | | -- Token usage and cost |
113 | | - |
114 | | -The output is meant to be reviewed and edited into a final report in `reports/`. |
115 | | - |
116 | | -## Scoring |
117 | | - |
118 | | -The scorer uses a two-stage filter: |
119 | | -1. **Fuzzy substring match** — checks if the perturbed text appears (approximately) in the review comment's quote, using normalized text coverage with a 0.75 threshold. |
120 | | -2. **LLM-as-judge** — asks a model to rate whether the reviewer's explanation identifies the same error described in the perturbation's `why_wrong` field (score >= 3/5 = match). |
121 | | - |
122 | | -## Known Limitations |
| 112 | +## Known limitations |
123 | 113 |
|
124 | | -- The perturb stage targets `n_per_error=2` perturbations per error type (8 total per paper), but the LLM often reuses the same candidate span for both, causing the validator to reject the duplicate. Typical yield is ~4-5 per paper. |
125 | | -- Fuzzy substring matching can miss catches where the reviewer heavily paraphrases the quoted text. |
126 | | -- `cost_usd` from OpenRouter metadata is unreliable for some models (notably qwen). |
| 114 | +- The generator often reuses the same candidate span for multiple errors, and the |
| 115 | + validator rejects the duplicates, so typical yield is ~4-5 perturbations per paper. |
| 116 | +- The substring match can miss detections where the reviewer heavily paraphrases |
| 117 | + the quoted text. |
| 118 | +- `cost_usd` from OpenRouter metadata is unreliable for some models. |
0 commit comments