Skip to content

Commit c641247

Browse files
author
Kevin-Li-2025
committed
Add reproducibility and CI polish
1 parent b9ff914 commit c641247

4 files changed

Lines changed: 419 additions & 11 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
test:
14+
name: Python tests
15+
runs-on: ubuntu-latest
16+
steps:
17+
- name: Check out repository
18+
uses: actions/checkout@v4
19+
20+
- name: Set up Python
21+
uses: actions/setup-python@v5
22+
with:
23+
python-version: "3.11"
24+
cache: pip
25+
26+
- name: Install package
27+
run: python -m pip install -e ".[dev,bench]"
28+
29+
- name: Run tests
30+
run: python -m pytest -q
31+
32+
- name: Build generalization scorecard
33+
run: |
34+
python scripts/build_generalization_scorecard.py \
35+
--output-dir /tmp/l20_codeforge_scorecard_ci
36+
37+
- name: Verify benchmark artifact hashes
38+
run: |
39+
python - <<'PY'
40+
import hashlib
41+
from pathlib import Path
42+
43+
expected = {
44+
"benchmarks/generalization_scorecard_2026_05_23/scorecard.json":
45+
"1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712",
46+
"benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json":
47+
"2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772",
48+
"benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv":
49+
"08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550",
50+
}
51+
for rel_path, expected_sha in expected.items():
52+
actual_sha = hashlib.sha256(Path(rel_path).read_bytes()).hexdigest()
53+
if actual_sha != expected_sha:
54+
raise SystemExit(f"{rel_path}: expected {expected_sha}, got {actual_sha}")
55+
print("benchmark artifact hashes verified")
56+
PY

‎README.md‎

Lines changed: 39 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# L20-CodeForge
22

3+
[![CI](https://github.com/Kevin-Li-2025/L20-CodeForge/actions/workflows/ci.yml/badge.svg)](https://github.com/Kevin-Li-2025/L20-CodeForge/actions/workflows/ci.yml)
4+
35
Single-L20 post-training, verifier-guided inference, and executable benchmark
46
infrastructure for code models.
57

@@ -13,6 +15,9 @@ Status: the repository currently demonstrates strong system-level gains on
1315
public benchmarks. It does not yet claim a +15 point greedy model-weight
1416
improvement over the base model.
1517

18+
Core docs: [Reproducibility](REPRODUCIBILITY.md) and
19+
[Architecture](docs/ARCHITECTURE.md).
20+
1621
## Results
1722

1823
| Benchmark | Base model / checkpoint | Protocol | Baseline | L20-CodeForge result | Delta | Artifact |
@@ -25,6 +30,17 @@ improvement over the base model.
2530
Cross-benchmark guardrail: `benchmarks/generalization_scorecard_2026_05_23/`
2631
records a `PASS` gate over full LiveCodeBench plus EvalPlus HumanEval+/MBPP+.
2732

33+
## Benchmark Artifact Hashes
34+
35+
| Artifact | SHA-256 |
36+
| --- | --- |
37+
| `benchmarks/generalization_scorecard_2026_05_23/scorecard.json` | `1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712` |
38+
| `benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json` | `2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772` |
39+
| `benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv` | `08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550` |
40+
41+
See [REPRODUCIBILITY.md](REPRODUCIBILITY.md) for the full hash-verification
42+
command and expected output.
43+
2844
## Claim Boundary
2945

3046
- The LiveCodeBench and EvalPlus numbers above are system-level results unless a
@@ -46,17 +62,7 @@ L20-CodeForge is organized around one constraint: a single NVIDIA L20 should be
4662
enough to run a serious coding post-training loop if the loop is selective,
4763
measured, and executable.
4864

49-
```mermaid
50-
flowchart LR
51-
A["Benchmark task"] --> B["Prompt and context builder"]
52-
B --> C["Candidate generation"]
53-
C --> D["Public tests and static health checks"]
54-
D --> E["Repair and feedback loop"]
55-
E --> F["Verifier or selector"]
56-
F --> G["Hidden-test replay"]
57-
G --> H["Scorecard and audit"]
58-
H --> I["SFT / DPO / RLVR data"]
59-
```
65+
![L20-CodeForge architecture](docs/assets/architecture.svg)
6066

6167
The repository contains:
6268

@@ -80,6 +86,12 @@ python -m l20_codeforge profile
8086
python -m l20_codeforge smoke-loop
8187
```
8288

89+
The `python -m pytest -q` line should print:
90+
91+
```text
92+
135 passed in <time>s
93+
```
94+
8395
On an L20 host:
8496

8597
```bash
@@ -102,6 +114,22 @@ python scripts/build_generalization_scorecard.py \
102114
--output-dir benchmarks/generalization_scorecard_2026_05_23
103115
```
104116

117+
Expected output includes:
118+
119+
```json
120+
{
121+
"status": "PASS",
122+
"checks": [
123+
{
124+
"name": "lcb_overall_improves",
125+
"value": 0.100474,
126+
"threshold": 0.0,
127+
"passed": true
128+
}
129+
]
130+
}
131+
```
132+
105133
Re-run the packaged EvalPlus scoring:
106134

107135
```bash

‎REPRODUCIBILITY.md‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Reproducibility
2+
3+
This document defines the exact local checks and benchmark-artifact verification
4+
used for the current L20-CodeForge README claims.
5+
6+
The repository keeps benchmark summaries, saved generations, evaluator outputs,
7+
and hashes under version control. Large private LiveCodeBench payloads are not
8+
committed; they must be materialized locally from the benchmark source when a
9+
full hidden replay is required.
10+
11+
## Environment
12+
13+
Recommended local setup:
14+
15+
```bash
16+
python3 -m venv .venv
17+
source .venv/bin/activate
18+
python -m pip install --upgrade pip
19+
python -m pip install -e ".[dev,bench]"
20+
```
21+
22+
Expected package install outcome:
23+
24+
```text
25+
Successfully installed l20-codeforge
26+
```
27+
28+
The GitHub Actions CI uses Python `3.11` and installs the same `.[dev,bench]`
29+
extras before running tests and scorecard checks.
30+
31+
## CI Check
32+
33+
Command:
34+
35+
```bash
36+
python -m pytest -q
37+
```
38+
39+
Expected output shape:
40+
41+
```text
42+
135 passed in <time>s
43+
```
44+
45+
The exact wall time depends on machine load. The pass count is the important
46+
invariant for this snapshot.
47+
48+
## Static L20 Profile
49+
50+
Command:
51+
52+
```bash
53+
python -m l20_codeforge profile
54+
```
55+
56+
Expected output contains:
57+
58+
```json
59+
{
60+
"gpu": "NVIDIA L20",
61+
"vram_gb": 48,
62+
"preferred_base_models": [
63+
"Qwen/Qwen2.5-Coder-7B-Instruct"
64+
]
65+
}
66+
```
67+
68+
## Generalization Scorecard
69+
70+
Command:
71+
72+
```bash
73+
python scripts/build_generalization_scorecard.py \
74+
--output-dir /tmp/l20_codeforge_scorecard_check
75+
```
76+
77+
Expected output:
78+
79+
```json
80+
{
81+
"status": "PASS",
82+
"checks": [
83+
{
84+
"name": "lcb_overall_improves",
85+
"value": 0.100474,
86+
"threshold": 0.0,
87+
"passed": true
88+
}
89+
]
90+
}
91+
```
92+
93+
The actual output includes all slice checks. For the committed snapshot, every
94+
check must have `"passed": true`.
95+
96+
## Benchmark Artifact Hashes
97+
98+
Verify the committed artifact hashes:
99+
100+
```bash
101+
shasum -a 256 \
102+
benchmarks/generalization_scorecard_2026_05_23/scorecard.json \
103+
benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json \
104+
benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv \
105+
benchmarks/livecodebench_full_release_v6_2026_05_22/qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json \
106+
benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json
107+
```
108+
109+
Expected output:
110+
111+
```text
112+
1eb0402378ea25732225b29d7ba367b6111ab3351e54cc7c01fa7646a7a12712 benchmarks/generalization_scorecard_2026_05_23/scorecard.json
113+
2a0ff919aa15eb9ecdf74824f7bf790a23f6d0197ef74970b6190c60e0e00772 benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json
114+
08732bbb76450f92ef3c02fa97a163aba01f71028365072c205c5a3af45d5550 benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv
115+
7272f5591c2f868c059226a2a5ec8fc772994cfafd20eb8397a2b6d90aed64bf benchmarks/livecodebench_full_release_v6_2026_05_22/qwen25_coder_7b_temp08_n8_public_select_full_eval/report.json
116+
e86db2af864a9c8896dcd1bc2d4d7b44af7fa395b856ea02b6f0e69c31c915cc benchmarks/evalplus_l20_codeforge_2026_05_22/rechecks/manifest.json
117+
```
118+
119+
The CI workflow verifies the three top-level claim artifacts on every push and
120+
pull request:
121+
122+
- `benchmarks/generalization_scorecard_2026_05_23/scorecard.json`
123+
- `benchmarks/livecodebench_full_release_v6_2026_05_22/full_n8_public_select_summary.json`
124+
- `benchmarks/evalplus_l20_codeforge_2026_05_22/summary.csv`
125+
126+
## LiveCodeBench Full Replay Boundary
127+
128+
The full hidden-test JSONL is intentionally not committed. To materialize it
129+
locally:
130+
131+
```bash
132+
python scripts/materialize_lcb_release_jsonl.py \
133+
--release-version release_v6 \
134+
--output-jsonl data/raw/livecodebench/full_release_v6/release_v6_test_full.jsonl \
135+
--manifest data/raw/livecodebench/full_release_v6/release_v6_test_full.manifest.json
136+
```
137+
138+
Expected committed summary for the current headline result:
139+
140+
```json
141+
{
142+
"benchmark": "livecodebench/code_generation_lite release_v6",
143+
"tasks": 1055,
144+
"score": {
145+
"passed": 403,
146+
"total": 1055,
147+
"pass_at_1": 0.3819905213270142
148+
},
149+
"baseline_greedy": {
150+
"passed": 297,
151+
"total": 1055,
152+
"pass_at_1": 0.28151658767772514
153+
}
154+
}
155+
```
156+
157+
The full command sequence for generation and hidden replay is documented in:
158+
159+
```text
160+
benchmarks/livecodebench_full_release_v6_2026_05_22/README.md
161+
```
162+
163+
## EvalPlus Rechecks
164+
165+
HumanEval+:
166+
167+
```bash
168+
python -m l20_codeforge eval-evalplus humaneval \
169+
benchmarks/evalplus_l20_codeforge_2026_05_22/samples/humaneval.mixed-target.literal-combined.public-consensus-selected.samples.jsonl \
170+
--output /tmp/humaneval_recheck.json \
171+
--parallel 8
172+
```
173+
174+
Expected committed result:
175+
176+
```text
177+
base pass@1 0.982
178+
plus pass@1 0.927
179+
```
180+
181+
MBPP+:
182+
183+
```bash
184+
python -m l20_codeforge eval-evalplus mbpp \
185+
benchmarks/evalplus_l20_codeforge_2026_05_22/samples/mbpp.temp08.n5-plus-basefallback-n30.public-consensus-shortest-selected.samples.jsonl \
186+
--output /tmp/mbpp_recheck.json \
187+
--parallel 8
188+
```
189+
190+
Expected committed result:
191+
192+
```text
193+
base pass@1 0.960
194+
plus pass@1 0.817
195+
```
196+
197+
## Clean-Room Claim Rules
198+
199+
- Public tests and public examples may be used for selection and repair.
200+
- Hidden/private tests may only be used for final replay and audit.
201+
- Do not report targeted probes as broad benchmark results.
202+
- Do not update benchmark headline claims without updating the scorecard and
203+
artifact hashes in this document.

0 commit comments

Comments
 (0)