Skip to content

Commit ecdf27c

Browse files
author
Kevin-Li-2025
committed
Add arXiv paper draft
1 parent c641247 commit ecdf27c

5 files changed

Lines changed: 467 additions & 3 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ artifacts/
1010
data/raw/
1111
data/processed/
1212
logs/
13+
paper/*.pdf
1314
*.safetensors
1415
*.pt
1516
*.pth
@@ -18,4 +19,3 @@ logs/
1819
*.ckpt
1920
.env
2021
.env.*
21-

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,8 @@ Status: the repository currently demonstrates strong system-level gains on
1515
public benchmarks. It does not yet claim a +15 point greedy model-weight
1616
improvement over the base model.
1717

18-
Core docs: [Reproducibility](REPRODUCIBILITY.md) and
19-
[Architecture](docs/ARCHITECTURE.md).
18+
Core docs: [Reproducibility](REPRODUCIBILITY.md),
19+
[Architecture](docs/ARCHITECTURE.md), and [paper draft](paper/README.md).
2020

2121
## Results
2222

‎paper/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Paper Draft
2+
3+
This directory contains the LaTeX draft for an arXiv-style paper about
4+
L20-CodeForge.
5+
6+
The draft is intentionally written as a measured research report. It claims
7+
system-level gains, not a new model checkpoint.
8+
9+
## Build
10+
11+
With Tectonic:
12+
13+
```bash
14+
cd paper
15+
tectonic main.tex
16+
```
17+
18+
With a standard TeX installation:
19+
20+
```bash
21+
cd paper
22+
pdflatex main.tex
23+
bibtex main
24+
pdflatex main.tex
25+
pdflatex main.tex
26+
```
27+
28+
## arXiv Notes
29+
30+
Before uploading source to arXiv:
31+
32+
- Submit TeX source, not a PDF generated from TeX.
33+
- Do not include generated files such as `.aux`, `.log`, `.out`, `.pdf`, or
34+
`.toc`.
35+
- Keep file names portable and case-consistent.
36+
- Use PDF/PNG/JPG figures when compiling with PDFLaTeX.
37+
- Verify title, abstract, authors, license, and category metadata in the arXiv
38+
submission UI.
39+
40+
Primary arXiv guidance:
41+
42+
- https://info.arxiv.org/help/submit/index.html
43+
- https://info.arxiv.org/help/submit_tex.html
44+
- https://info.arxiv.org/help/license/index.html

‎paper/main.tex‎

Lines changed: 318 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,318 @@
1+
\documentclass[11pt]{article}
2+
3+
\usepackage[margin=1in]{geometry}
4+
\usepackage{amsmath}
5+
\usepackage{booktabs}
6+
\usepackage{float}
7+
\usepackage{graphicx}
8+
\usepackage{microtype}
9+
\usepackage[numbers,sort&compress]{natbib}
10+
\usepackage[colorlinks=true,linkcolor=black,citecolor=black,urlcolor=blue]{hyperref}
11+
\usepackage{tikz}
12+
\usetikzlibrary{arrows.meta,positioning,fit}
13+
14+
\title{L20-CodeForge: Auditable Test-Time Scaling for Code Generation on a Single GPU}
15+
\author{Anonymous Authors}
16+
\date{May 2026}
17+
18+
\begin{document}
19+
\maketitle
20+
21+
\begin{abstract}
22+
Large code models are often evaluated in settings that assume substantial
23+
serving budgets, broad sampling, and opaque benchmark pipelines. This paper
24+
studies a narrower question: how much reliable coding performance can be
25+
extracted from a 7B-class open model when the compute budget is a single NVIDIA
26+
L20 GPU and hidden tests are reserved for measurement. We describe
27+
L20-CodeForge, an auditable post-training and inference stack for code
28+
generation. The system separates public-side candidate generation, public-test
29+
selection, repair, and verifier experiments from private hidden-test replay. On
30+
the full 1,055-task LiveCodeBench \texttt{release\_v6} code-generation suite,
31+
Qwen2.5-Coder-7B-Instruct improves from 297 solved tasks under greedy decoding
32+
to 403 solved tasks with eight-sample public-test selection, a gain of 10.05
33+
percentage points. On EvalPlus, the clean public-signal system improves
34+
HumanEval+ from 84.8\% to 92.7\% and MBPP+ from 72.2\% to 81.7\%. We also
35+
report negative results: repeated public-feedback repair can overfit visible
36+
tests, input-only differential tests do little without multiple
37+
public-passing candidates, and an uncalibrated expected-output verifier can
38+
regress hidden replay. The main claim is not that a new model checkpoint
39+
outperforms the base model; rather, the result is a reproducible case study in
40+
benchmark hygiene, test-time scaling, and failure-aware post-training
41+
infrastructure under a small-GPU constraint.
42+
\end{abstract}
43+
44+
\section{Introduction}
45+
46+
Execution is both the advantage and the trap of code generation. It gives a
47+
cheap verifier for many tasks, but it also creates easy ways to overstate
48+
progress: select on visible examples, tune against stale hidden tests, or report
49+
small optimistic subsets. This tension is sharper for resource-constrained
50+
post-training. A single accelerator cannot support the search budgets or
51+
training sweeps used by large labs, so every evaluation decision carries more
52+
weight.
53+
54+
This paper reports a small-GPU study rather than a new frontier model. We use a
55+
single NVIDIA L20 as the target hardware and ask what can be gained by improving
56+
the surrounding system: candidate generation, public-test selection, repair,
57+
candidate health auditing, hidden replay, and cross-benchmark scorecards. The
58+
model in the main experiment is Qwen2.5-Coder-7B-Instruct. Hidden LiveCodeBench
59+
tests are used only for final replay and audit, not for candidate selection or
60+
repair.
61+
62+
The study makes four contributions.
63+
64+
\begin{enumerate}
65+
\item We define an auditable single-GPU evaluation protocol for code
66+
generation with a strict public/private boundary.
67+
\item We report full-suite LiveCodeBench \texttt{release\_v6} and EvalPlus
68+
results with committed artifacts, hashes, and re-run commands.
69+
\item We analyze several verifier and repair variants, including negative
70+
results that would be hidden by a headline-only report.
71+
\item We release the supporting repository as a reproducibility artifact
72+
with CI, scorecards, benchmark summaries, and exact hash checks.
73+
\end{enumerate}
74+
75+
\section{Related Work}
76+
77+
\paragraph{Code-generation benchmarks.}
78+
LiveCodeBench was introduced to reduce contamination and broaden code-model
79+
evaluation beyond static HumanEval-style tasks, including code generation,
80+
self-repair, code execution, and test-output prediction
81+
\citep{jain2024livecodebench}. EvalPlus showed that common code benchmarks can
82+
miss incorrect programs because their tests are insufficient; it augments
83+
HumanEval and MBPP with many additional tests and can change model rankings
84+
\citep{liu2023evalplus}. These benchmarks motivate the two evaluation choices
85+
used here: full-suite replay on LiveCodeBench and cross-checking with EvalPlus.
86+
87+
\paragraph{Test-time scaling and execution-guided selection.}
88+
AlphaCode demonstrated that large-scale sampling followed by program-behavior
89+
filtering is central to competitive-programming performance
90+
\citep{li2022alphacode}. CodeT generates tests and uses execution agreement to
91+
select among samples \citep{chen2022codet}. LEVER trains an execution-aware
92+
verifier to rerank generated programs \citep{ni2023lever}. S* studies
93+
test-time scaling for code and reports large gains from hybrid search and
94+
selection on LiveCodeBench \citep{wang2025sstar}. L20-CodeForge follows the
95+
same broad line of work but restricts the compute budget and emphasizes
96+
auditable artifacts over maximum score.
97+
98+
\paragraph{Verified data and code RL.}
99+
Recent code-post-training work shows that verified competitive-programming data
100+
and high-quality tests can substantially improve small models. rStar-Coder
101+
constructs verified reasoning data and reports large LiveCodeBench gains for
102+
Qwen2.5 models \citep{guan2025rstarcoder}. HardTests studies high-quality test
103+
synthesis for difficult coding problems and finds that stronger tests improve
104+
both evaluation and training signal \citep{he2025hardtests}. X-Coder reports a
105+
Qwen2.5-derived RLVR model trained with verified code-reasoning data
106+
\citep{xcode2026xcoder}. These results suggest that durable model-weight gains
107+
need verified data and calibrated rewards, not only prompt changes.
108+
109+
\section{Protocol}
110+
111+
\subsection{Public and Private Signals}
112+
113+
Each benchmark problem is treated as two objects. The public object contains the
114+
prompt, starter code when available, visible examples, and public tests. It may
115+
be used for candidate generation, public-test selection, and repair. The private
116+
object contains hidden tests. It is used only for final replay and audit.
117+
118+
The protocol intentionally permits public-test selection because public tests
119+
are available to any solver. It does not permit hidden-test selection, hidden
120+
failure feedback, or retuning against hidden outcomes. Negative outcomes are
121+
kept in the artifact record because they identify where public-side methods fail
122+
to generalize.
123+
124+
\subsection{Hardware and Model}
125+
126+
The target hardware is a single NVIDIA L20 GPU. The main LiveCodeBench and
127+
EvalPlus experiments use Qwen2.5-Coder-7B-Instruct without an adapter. The
128+
reported LiveCodeBench system result uses temperature 0.8, top-p 0.95, eight
129+
samples per task, public-test selection, and hidden-test replay.
130+
131+
\section{System}
132+
133+
Figure~\ref{fig:system} summarizes the implementation. Candidate generation and
134+
repair are public-side operations. Hidden replay is outside the selection loop.
135+
The scorecard combines LiveCodeBench and EvalPlus to catch benchmark-specific
136+
overfitting.
137+
138+
\begin{figure}[H]
139+
\centering
140+
\begin{tikzpicture}[
141+
node distance=7mm and 10mm,
142+
box/.style={draw, rounded corners, align=center, minimum height=9mm, minimum width=24mm, font=\small},
143+
public/.style={box, fill=blue!6, draw=blue!55!black},
144+
verify/.style={box, fill=green!7, draw=green!45!black},
145+
audit/.style={box, fill=orange!10, draw=orange!70!black},
146+
train/.style={box, fill=violet!7, draw=violet!60!black},
147+
arrow/.style={-Latex, thick}
148+
]
149+
\node[public] (task) {Benchmark\\task};
150+
\node[public, right=of task] (context) {Context\\builder};
151+
\node[public, right=of context] (gen) {Candidate\\generation};
152+
\node[verify, right=of gen] (publictest) {Public tests\\and health};
153+
\node[verify, below=of publictest] (repair) {Repair\\feedback};
154+
\node[verify, left=of repair] (selector) {Selector or\\verifier};
155+
\node[audit, below=of selector] (hidden) {Hidden-test\\replay};
156+
\node[audit, right=of hidden] (score) {Scorecard\\and hashes};
157+
\node[audit, right=of score] (audit) {Failure\\audit};
158+
\node[train, below=of hidden] (store) {Trajectory\\store};
159+
\node[train, right=of store] (data) {SFT, DPO,\\RLVR data};
160+
\node[train, right=of data] (adapter) {Small adapter\\or verifier};
161+
162+
\draw[arrow] (task) -- (context);
163+
\draw[arrow] (context) -- (gen);
164+
\draw[arrow] (gen) -- (publictest);
165+
\draw[arrow] (publictest) -- (repair);
166+
\draw[arrow] (repair) -- (selector);
167+
\draw[arrow] (selector) -- (hidden);
168+
\draw[arrow] (hidden) -- (score);
169+
\draw[arrow] (score) -- (audit);
170+
\draw[arrow] (audit) -- (adapter);
171+
\draw[arrow] (adapter) -- (data);
172+
\draw[arrow] (data) -- (store);
173+
\draw[arrow, dashed] (store) -- (context);
174+
\end{tikzpicture}
175+
\caption{L20-CodeForge evaluation loop. Public signals drive generation,
176+
selection, and repair. Private tests are reserved for final replay and audits.}
177+
\label{fig:system}
178+
\end{figure}
179+
180+
\section{Experiments}
181+
182+
\subsection{LiveCodeBench}
183+
184+
Table~\ref{tab:lcb-main} reports the main full-suite LiveCodeBench result. The
185+
greedy baseline solves 297 of 1,055 tasks. Four-sample public-test selection
186+
solves 378 tasks. Eight-sample selection solves 403 tasks, an absolute gain of
187+
106 tasks over greedy decoding.
188+
189+
\begin{table}[H]
190+
\centering
191+
\begin{tabular}{llrrrr}
192+
\toprule
193+
Model & Protocol & Samples & Solved & Total & Pass@1 \\
194+
\midrule
195+
Qwen2.5-Coder-7B-Instruct & greedy & 1 & 297 & 1055 & 28.15 \\
196+
Qwen2.5-Coder-7B-Instruct & public selection & 4 & 378 & 1055 & 35.83 \\
197+
Qwen2.5-Coder-7B-Instruct & public selection & 8 & 403 & 1055 & 38.20 \\
198+
\bottomrule
199+
\end{tabular}
200+
\caption{LiveCodeBench \texttt{release\_v6} full-suite results. Public
201+
selection uses only visible public tests; hidden tests are used for final replay.}
202+
\label{tab:lcb-main}
203+
\end{table}
204+
205+
The gain is not uniform across difficulty levels. Easy tasks improve from 206
206+
to 260 solved tasks, medium tasks from 82 to 124, and hard tasks from 9 to 19.
207+
The hard-task result remains weak in absolute terms, which is consistent with
208+
the need for stronger algorithmic candidates rather than only better selection.
209+
210+
\begin{table}[H]
211+
\centering
212+
\begin{tabular}{lrrrrr}
213+
\toprule
214+
Slice & Greedy solved & System solved & Total & Greedy & System \\
215+
\midrule
216+
Easy & 206 & 260 & 322 & 63.98 & 80.75 \\
217+
Medium & 82 & 124 & 383 & 21.41 & 32.38 \\
218+
Hard & 9 & 19 & 350 & 2.57 & 5.43 \\
219+
\bottomrule
220+
\end{tabular}
221+
\caption{LiveCodeBench difficulty breakdown. Values in the last two columns are
222+
percentages.}
223+
\label{tab:lcb-breakdown}
224+
\end{table}
225+
226+
\subsection{EvalPlus}
227+
228+
Table~\ref{tab:evalplus} reports the EvalPlus guardrail. The clean system rows
229+
use public prompt and base-test signals, but not EvalPlus extra tests, for
230+
selection. Both HumanEval+ and MBPP+ improve over greedy decoding.
231+
232+
\begin{table}[H]
233+
\centering
234+
\begin{tabular}{lrrrr}
235+
\toprule
236+
Dataset & Greedy base & System base & Greedy plus & System plus \\
237+
\midrule
238+
HumanEval & 89.0 & 98.2 & 84.8 & 92.7 \\
239+
MBPP & 82.8 & 96.0 & 72.2 & 81.7 \\
240+
\bottomrule
241+
\end{tabular}
242+
\caption{EvalPlus results. Values are pass@1 percentages. The plus columns use
243+
the stricter EvalPlus tests for scoring, not for candidate selection.}
244+
\label{tab:evalplus}
245+
\end{table}
246+
247+
\subsection{X-Coder Control Slice}
248+
249+
We also probed IIGroup/X-Coder-RL-Qwen2.5-7B on a medium-difficulty control
250+
slice. Automatic and strict starter-prefix checks both scored 0/12. Public-only
251+
second-pass repair reached 2/12. A single public-feedback repair round reached
252+
4/12. A second public-feedback round produced visible-test signal but no hidden
253+
passes. We therefore treat the first feedback round as useful but not sufficient
254+
for broad claims.
255+
256+
\section{Failure Analysis}
257+
258+
The main failure class on the eight-sample LiveCodeBench run is wrong answer,
259+
not syntax or missing entrypoints. This suggests that future work should
260+
prioritize better algorithmic candidates and stronger verifier calibration over
261+
format-only prompting.
262+
263+
Several attempted improvements did not hold up under hidden replay. Repeated
264+
public-feedback repair overfit visible tests on the second round. Input-only
265+
differential fuzzing improved candidate-pair coverage on some targets but did
266+
not change selection when the pool contained too few public-passing alternatives. An
267+
expected-output verifier over generated inputs regressed the targeted replay,
268+
indicating that verifier confidence needs calibration before it can safely alter
269+
headline selections.
270+
271+
\section{Reproducibility}
272+
273+
The repository includes committed benchmark summaries, saved generations,
274+
reports, hash manifests, and CI checks. The full hidden-test LiveCodeBench JSONL
275+
is not committed. Instead, the repository records materialization commands,
276+
source hashes, generation hashes, evaluator outputs, and compact summaries.
277+
278+
For the main LiveCodeBench result, the committed summary hash is:
279+
280+
\begin{center}
281+
\small
282+
\texttt{2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772}
283+
\end{center}
284+
285+
The generalization scorecard hash is:
286+
287+
\begin{center}
288+
\small
289+
\texttt{1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712}
290+
\end{center}
291+
292+
The repository also includes a separate reproducibility document with expected
293+
command outputs and artifact-hash checks.
294+
295+
\section{Limitations}
296+
297+
The strongest limitation is that the main gain is a system-level gain, not a
298+
model-weight gain. The eight-sample selector is useful in practice but should
299+
not be confused with a greedy checkpoint improvement. The experiments also use
300+
one main base model and one main GPU type. Public tests are legitimate solver
301+
signals, but repeated repair on public failures can overfit them. Finally,
302+
private benchmark tests cannot be committed directly, so full hidden replay
303+
requires local materialization of the benchmark payload.
304+
305+
\section{Conclusion}
306+
307+
L20-CodeForge shows that a careful single-GPU system can extract a meaningful
308+
amount of additional coding performance from a 7B open model while preserving a
309+
clean benchmark boundary. The useful parts are not only the positive results,
310+
but also the audit trail: public selection helps, first-round repair can help,
311+
and several plausible verifier variants fail without calibration. The next step
312+
is to convert this infrastructure into a verified-data and RLVR pipeline that
313+
improves single-sample model behavior on held-out tasks.
314+
315+
\bibliographystyle{plainnat}
316+
\bibliography{references}
317+
318+
\end{document}

0 commit comments

Comments
 (0)