Refine arXiv paper positioning

Author: Kevin-Li-2025

paper/ARXIV_SUBMISSION.md

@@ -42,10 +42,10 @@ Rationale:
 Suggested text:
 
 ```text
-Preprint. Code, benchmark artifacts, CI, and reproducibility notes are available at https://github.com/Kevin-Li-2025/L20-CodeForge.
+10 pages. Code, benchmark artifacts, CI, and reproducibility notes are available at https://github.com/Kevin-Li-2025/L20-CodeForge.
 ```
 
-Update the page count after the final compile.
+Update the page count again if the final compile changes.
 
 ## License
 

paper/README.md

@@ -41,6 +41,10 @@ Metadata and category plan:
 
 - See `ARXIV_SUBMISSION.md`.
 
+Reference scope and source checks:
+
+- See `REFERENCE_AUDIT.md`.
+
 Source package:
 
 ```bash

paper/REFERENCE_AUDIT.md

@@ -0,0 +1,45 @@
+# Reference Audit
+
+This note records why the paper cites each research line and how the entries
+were checked. The bibliography is intentionally selective: it covers the
+closest benchmark, verifier, code-RL, verified-data, and feedback-loop work
+without padding the paper with loosely related citations.
+
+## Coverage
+
+| Area | References | Why they are included |
+| --- | --- | --- |
+| Code benchmark hygiene | LiveCodeBench, EvalPlus | Defines the contamination and test-adequacy risks that motivate full-suite hidden replay and EvalPlus guardrails. |
+| Test-time scaling | AlphaCode, CodeT, LEVER, S* | Covers sampling, generated tests, execution-aware verifiers, and recent code-specific test-time scaling. |
+| Code RL and reward modeling | CodeRL, ACECoder | Covers unit-test/critic feedback, test-case synthesis, reward modeling, and RL for code models. |
+| Verified synthetic data | rStar-Coder, HardTests, X-Coder | Covers the closest recent direction for durable model-weight gains from verified tasks, tests, and solutions. |
+| Feedback/refinement loops | OpenCodeInterpreter, Reflexion | Covers execution feedback, refinement, and verbal reflection; this anchors the paper's negative repair-loop findings. |
+| Base model | Qwen2.5-Coder technical report | Identifies the open 7B model used in the main experiments. |
+
+## Primary Sources Checked
+
+- LiveCodeBench: https://arxiv.org/abs/2403.07974
+- EvalPlus: https://arxiv.org/abs/2305.01210
+- AlphaCode: https://arxiv.org/abs/2203.07814
+- CodeT: https://arxiv.org/abs/2207.10397
+- LEVER: https://arxiv.org/abs/2302.08468
+- S*: https://arxiv.org/abs/2502.14382
+- CodeRL: https://arxiv.org/abs/2207.01780
+- ACECoder: https://arxiv.org/abs/2502.01718
+- rStar-Coder: https://arxiv.org/abs/2505.21297
+- HardTests: https://arxiv.org/abs/2505.24098
+- X-Coder: https://arxiv.org/abs/2601.06953
+- OpenCodeInterpreter: https://arxiv.org/abs/2402.14658
+- Reflexion: https://arxiv.org/abs/2303.11366
+- Qwen2.5-Coder: https://qwenlm.github.io/blog/qwen2.5-coder-family/
+
+## Submission Sources
+
+The arXiv and checklist sources are used for packaging and metadata, not cited
+as research contributions in the paper body:
+
+- arXiv submission overview: https://info.arxiv.org/help/submit/index.html
+- arXiv TeX submission help: https://info.arxiv.org/help/submit_tex.html
+- arXiv license help: https://info.arxiv.org/help/license/index.html
+- arXiv category taxonomy: https://arxiv.org/category_taxonomy
+- NeurIPS paper checklist: https://nips.cc/public/guides/PaperChecklist

paper/main.tex

@@ -97,7 +97,7 @@ \section{Claim Boundary}
 or cross-benchmark checks. These failures are reported because they define the
 boundary between useful test-time scaling and visible-test overfitting.
 
-\section{Related Work}
+\section{Related Work and Positioning}
 
 \paragraph{Code-generation benchmarks.}
 LiveCodeBench was introduced to reduce contamination and broaden code-model
@@ -121,30 +121,73 @@ \section{Related Work}
 select among samples \citep{chen2022codet}. LEVER trains an execution-aware
 verifier to rerank generated programs \citep{ni2023lever}. S* studies
 test-time scaling for code and reports large gains from hybrid search and
-selection on LiveCodeBench \citep{wang2025sstar}. L20-CodeForge follows the
+selection on LiveCodeBench \citep{li2025sstar}. L20-CodeForge follows the
 same broad line of work but restricts the compute budget and emphasizes
 auditable artifacts over maximum score.
 
 \paragraph{Verified data and code RL.}
-Recent code-post-training work shows that verified competitive-programming data
-and high-quality tests can substantially improve small models. rStar-Coder
-constructs verified reasoning data and reports large LiveCodeBench gains for
-Qwen2.5 models \citep{guan2025rstarcoder}. HardTests studies high-quality test
-synthesis for difficult coding problems and finds that stronger tests improve
-both evaluation and training signal \citep{he2025hardtests}. X-Coder reports a
-Qwen2.5-derived RLVR model trained with verified code-reasoning data
-\citep{xcode2026xcoder}. These results suggest that durable model-weight gains
-need verified data and calibrated rewards, not only prompt changes.
-
-\paragraph{Reproducible ML reporting.}
-The paper structure follows the reproducibility concerns emphasized by arXiv
-submission guidance and the NeurIPS checklist: state compute, data boundaries,
-evaluation protocol, limitations, and artifacts that allow independent
-inspection \citep{arxiv2026submit,neurips2026checklist}. This is especially
-important for executable code benchmarks, where small changes in candidate
-selection can change the apparent result.
-
-\section{Method}
+CodeRL uses unit-test and critic feedback for reinforcement learning and
+test-time regeneration \citep{le2022coderl}. ACECoder synthesizes test cases to
+train reward models and run code RL, reporting gains across HumanEval, MBPP,
+BigCodeBench, and LiveCodeBench \citep{zeng2025acecoder}. Recent
+post-training work also shows that verified competitive-programming data and
+high-quality tests can substantially improve small models. rStar-Coder
+constructs a large verified reasoning dataset and reports large LiveCodeBench
+gains for Qwen2.5 models \citep{liu2025rstarcoder}. HardTests studies
+high-quality test synthesis for difficult coding problems and finds that
+stronger tests improve both evaluation and training signal
+\citep{he2025hardtests}. X-Coder reports a Qwen2.5-derived RLVR model trained
+with fully synthetic tasks, solutions, and tests \citep{wu2026xcoder}. These
+results suggest that durable model-weight gains need verified data and
+calibrated rewards, not only prompt changes.
+
+\paragraph{Execution feedback and refinement.}
+OpenCodeInterpreter integrates code generation with execution and refinement
+through multi-turn feedback \citep{zheng2024opencodeinterpreter}. Reflexion
+uses verbal feedback to improve language-agent behavior across tasks including
+coding \citep{shinn2023reflexion}. L20-CodeForge differs in emphasis: the
+repair loop is not treated as an unconstrained self-improvement process. It is
+bounded by a public/private evaluation contract, and repeated feedback is
+reported as a possible overfitting source when hidden replay does not improve.
+
+\begin{table}[H]
+\centering
+\small
+\begin{tabular}{>{\raggedright\arraybackslash}p{0.25\linewidth}
+                >{\raggedright\arraybackslash}p{0.30\linewidth}
+                >{\raggedright\arraybackslash}p{0.34\linewidth}}
+\toprule
+Line of work & Main idea & Difference in this paper \\
+\midrule
+AlphaCode, CodeT, LEVER, S* & Sampling, execution agreement, verifier or
+test-time selection & Same family of ideas, but constrained to one L20 and
+reported with full artifact hashes and negative results \\
+CodeRL, ACECoder & RL or reward modeling from unit tests and synthesized tests
+& Treated as the next model-weight direction; the present claim remains
+inference-system level \\
+rStar-Coder, HardTests, X-Coder & Verified synthetic tasks, tests, and
+reasoning data for stronger code models & Used as evidence that verified data
+is required for durable single-sample gains \\
+OpenCodeInterpreter, Reflexion & Execution/reflection feedback loops for code
+or agents & This paper caps public-feedback repair and shows when repeated
+feedback overfits visible tests \\
+LiveCodeBench, EvalPlus & Stronger contamination and test adequacy standards
+for code evaluation & Used as measurement infrastructure and as guardrails
+against optimistic subset results \\
+\bottomrule
+\end{tabular}
+\caption{Positioning against the closest research lines.}
+\label{tab:positioning}
+\end{table}
+
+Taken together, this literature suggests a conservative boundary for the
+present work. Public-test selection is a legitimate test-time system
+intervention, but durable model-weight claims require verified training data,
+calibrated reward signals, and held-out single-sample gains. The experiments
+below are therefore reported as an auditable inference and post-training
+infrastructure result, not as evidence that the base checkpoint has improved.
+
+\section{Design and Protocol}
 
 \subsection{Public and Private Signals}
 
@@ -234,7 +277,7 @@ \subsection{Hardware and Model}
 reported LiveCodeBench system result uses temperature 0.8, top-p 0.95, eight
 samples per task, public-test selection, and hidden-test replay.
 
-\section{System}
+\section{System Modules}
 
 Figure~\ref{fig:system} summarizes the implementation. Candidate generation and
 repair are public-side operations. Hidden replay is outside the selection loop.
@@ -283,6 +326,16 @@ \section{System}
 \label{fig:system}
 \end{figure}
 
+The system is organized so each stage can be rerun without changing the
+contract of the adjacent stages. The context builder emits public-only prompt
+records. The generator emits immutable candidate files. The selector consumes
+candidate files and public execution outcomes, but never private labels. Hidden
+replay consumes exactly one selected candidate per task. The scorecard then
+joins benchmark summaries, hashes, and failure counts into a read-only report.
+This separation is a practical guardrail: a new repair policy, verifier, or
+adapter can be swapped into the public side while preserving the same hidden
+replay and scorecard path.
+
 \section{Experiments}
 
 \subsection{LiveCodeBench}

paper/references.bib

@@ -1,5 +1,5 @@
 @misc{jain2024livecodebench,
-  title = {LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code},
+  title = {{LiveCodeBench}: Holistic and Contamination Free Evaluation of Large Language Models for Code},
   author = {Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion},
   year = {2024},
   eprint = {2403.07974},
@@ -8,7 +8,7 @@ @misc{jain2024livecodebench
 }
 
 @misc{liu2023evalplus,
-  title = {Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
+  title = {Is Your Code Generated by {ChatGPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation},
   author = {Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming},
   year = {2023},
   eprint = {2305.01210},
@@ -17,7 +17,7 @@ @misc{liu2023evalplus
 }
 
 @article{li2022alphacode,
-  title = {Competition-Level Code Generation with AlphaCode},
+  title = {Competition-Level Code Generation with {AlphaCode}},
   author = {Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, Remi and Eccles, Tom and Keeling, James and Gimeno, Felix and Dal Lago, Agustin and Hubert, Thomas and Choy, Peter and d'Autume, Cyprien de Masson and Babuschkin, Igor and Chen, Xinyun and Huang, Po-Sen and Welbl, Johannes and Gowal, Sven and Cherepanov, Alexey and Molloy, James and Mankowitz, Daniel J. and Robson, Esme Sutherland and Kohli, Pushmeet and de Freitas, Nando and Kavukcuoglu, Koray and Vinyals, Oriol},
   journal = {Science},
   year = {2022},
@@ -27,7 +27,7 @@ @article{li2022alphacode
 }
 
 @misc{chen2022codet,
-  title = {CodeT: Code Generation with Generated Tests},
+  title = {{CodeT}: Code Generation with Generated Tests},
   author = {Chen, Bei and Zhang, Fengji and Nguyen, Anh and Zan, Daoguang and Lin, Zeqi and Lou, Jian-Guang and Chen, Weizhu},
   year = {2022},
   eprint = {2207.10397},
@@ -36,25 +36,43 @@ @misc{chen2022codet
 }
 
 @misc{ni2023lever,
-  title = {LEVER: Learning to Verify Language-to-Code Generation with Execution},
+  title = {{LEVER}: Learning to Verify Language-to-Code Generation with Execution},
   author = {Ni, Ansong and Iyer, Srini and Radev, Dragomir and Stoyanov, Ves and Yih, Wen-tau and Wang, Sida I. and Lin, Xi Victoria},
   year = {2023},
   eprint = {2302.08468},
   archivePrefix = {arXiv},
   primaryClass = {cs.CL}
 }
 
-@misc{wang2025sstar,
-  title = {S*: Test Time Scaling for Code Generation},
+@misc{li2025sstar,
+  title = {{S*}: Test Time Scaling for Code Generation},
   author = {Li, Dacheng and Cao, Shiyi and Cao, Chengkun and Li, Xiuyu and Tan, Shangyin and Keutzer, Kurt and Xing, Jiarong and Gonzalez, Joseph E. and Stoica, Ion},
   year = {2025},
   eprint = {2502.14382},
   archivePrefix = {arXiv},
-  primaryClass = {cs.CL}
+  primaryClass = {cs.LG}
 }
 
-@misc{guan2025rstarcoder,
-  title = {rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset},
+@misc{le2022coderl,
+  title = {{CodeRL}: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning},
+  author = {Le, Hung and Wang, Yue and Gotmare, Akhilesh Deepak and Savarese, Silvio and Hoi, Steven C. H.},
+  year = {2022},
+  eprint = {2207.01780},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.LG}
+}
+
+@misc{zeng2025acecoder,
+  title = {{ACECODER}: Acing Coder {RL} via Automated Test-Case Synthesis},
+  author = {Zeng, Huaye and Jiang, Dongfu and Wang, Haozhe and Nie, Ping and Chen, Xiaotong and Chen, Wenhu},
+  year = {2025},
+  eprint = {2502.01718},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.SE}
+}
+
+@misc{liu2025rstarcoder,
+  title = {{rStar-Coder}: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset},
   author = {Liu, Yifei and Zhang, Li Lyna and Zhu, Yi and Dong, Bingcheng and Zhou, Xudong and Shang, Ning and Yang, Fan and Yang, Mao},
   year = {2025},
   eprint = {2505.21297},
@@ -63,40 +81,44 @@ @misc{guan2025rstarcoder
 }
 
 @misc{he2025hardtests,
-  title = {HardTests: Synthesizing High-Quality Test Cases for LLM Coding},
+  title = {{HardTests}: Synthesizing High-Quality Test Cases for {LLM} Coding},
   author = {He, Zhongmou and Choi, Yee Man and Zhang, Kexun and Ji, Jiabao and Zhou, Junting and Xu, Dejia and Bercovich, Ivan and Zhang, Aidan and Li, Lei},
   year = {2025},
   eprint = {2505.24098},
   archivePrefix = {arXiv},
-  primaryClass = {cs.SE}
+  primaryClass = {cs.CL}
 }
 
-@misc{xcode2026xcoder,
-  title = {X-Coder: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests},
+@misc{wu2026xcoder,
+  title = {{X-Coder}: Advancing Competitive Programming with Fully Synthetic Tasks, Solutions, and Tests},
   author = {Wu, Jie and Li, Haoling and Zhang, Xin and Guo, Jiani and Luo, Jane and Liu, Steven and Huang, Yangyu and Chu, Ruihang and Li, Scarlett and Yang, Yujiu},
   year = {2026},
   eprint = {2601.06953},
   archivePrefix = {arXiv},
   primaryClass = {cs.CL}
 }
 
-@misc{qwen2024coder,
-  title = {Qwen2.5-Coder Technical Report},
-  author = {{Qwen Team}},
+@misc{zheng2024opencodeinterpreter,
+  title = {{OpenCodeInterpreter}: Integrating Code Generation with Execution and Refinement},
+  author = {Zheng, Tianyu and Zhang, Ge and Shen, Tianhao and Liu, Xueling and Lin, Bill Yuchen and Fu, Jie and Chen, Wenhu and Yue, Xiang},
   year = {2024},
-  howpublished = {\url{https://qwenlm.github.io/blog/qwen2.5-coder-family/}}
+  eprint = {2402.14658},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.SE}
 }
 
-@misc{arxiv2026submit,
-  title = {arXiv Submission Guidelines},
-  author = {{arXiv}},
-  year = {2026},
-  howpublished = {\url{https://info.arxiv.org/help/submit/index.html}}
+@misc{shinn2023reflexion,
+  title = {{Reflexion}: Language Agents with Verbal Reinforcement Learning},
+  author = {Shinn, Noah and Cassano, Federico and Berman, Edward and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu},
+  year = {2023},
+  eprint = {2303.11366},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.AI}
 }
 
-@misc{neurips2026checklist,
-  title = {NeurIPS Paper Checklist Guidelines},
-  author = {{NeurIPS}},
-  year = {2026},
-  howpublished = {\url{https://nips.cc/public/guides/PaperChecklist}}
+@misc{qwen2024coder,
+  title = {{Qwen2.5-Coder} Technical Report},
+  author = {{Qwen Team}},
+  year = {2024},
+  howpublished = {\url{https://qwenlm.github.io/blog/qwen2.5-coder-family/}}
 }