Skip to content

Commit 9ec20b9

Browse files
john-b-yangjohn-b-yang
authored
Add SCML OneShot CC:Ladder (#113)
* Add SCML OneShot CC:Ladder (arena wiring + 51 human bots + build tooling) - Wire SCML for the ladder: SCML.Dockerfile now git-clones CodeClash-ai/SCML (seeded with the runtime) so branch_init works, matching the other arenas. - 51 human bots imported to CodeClash-ai/SCML human/* branches (source of truth, not committed here): 3 built-in baselines + 48 ANAC agents (2021-2024) ported from yasserfarouk/scml-agents into the arena's decide(observation) contract. RL/learned agents ported heuristic-core best-effort; scml2023/team_139 deferred. - configs/ablations/ladder/make_scml.yaml: round-robin (400 sims) for Elo ranking; README note. - scripts/ladder/: operational build tooling (porting guide, validators, smoke/push scripts, AWS runbook, example agents); ports/ regenerated, not committed. - docs/scml_game_visualization.html: explainer of the SCML game. * Add lock for image construction * Fix pre-commit issues * Update win conditions * Dockerfile update * update elo to include include-round-0 flag for ladder construction * bump timeout * Remove scripts/ladder folder * Remove visualization
1 parent fc4b60e commit 9ec20b9

4 files changed

Lines changed: 194 additions & 32 deletions

File tree

‎.claude/skills/create-ladder/SKILL.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -104,9 +104,14 @@ Local validation is necessary but NOT sufficient — a local shim skips Docker,
104104
`GITHUB_TOKEN=$(gh auth token) uv run codeclash ladder make configs/ablations/ladder/make_<a>.yaml --workers N`
105105
→ logs land under `logs/ladder/<A>/`. Fast arenas run on a laptop; big ones (e.g. SCML's
106106
~1275 pairs) want an AWS box under `tmux`/`nohup`.
107-
4. Rank: `python -m codeclash.analysis.metrics.elo -d logs/ladder/<A> --output-dir assets/<a>_elo`
107+
4. Rank: `uv run python -m codeclash.analysis.metrics.elo -d logs/ladder/<A> --include-round-0 --output-dir assets/<a>_elo`
108108
→ prints the Bradley-Terry/Elo order (weakest → strongest); that ordering IS the ladder.
109-
Tip: run a cheap low-`sims` pilot first and eyeball that baselines sit near the bottom.
109+
- **`--include-round-0` is REQUIRED for ladder construction.** `ladder make` uses
110+
`tournament.rounds: 0`, so round 0 IS the match; without the flag every tournament is
111+
dropped (round 0 is normally the excluded identical-codebases baseline) and the fit
112+
crashes on an empty matrix. Do NOT pass it for normal multi-round PvP/climbing Elo.
113+
- Use `uv run python`, not bare `python` — the analysis deps (matplotlib) live in the uv venv.
114+
- Tip: run a cheap low-`sims` pilot first and eyeball that baselines sit near the bottom.
110115

111116
## Phase 5 — Assemble the ranked configs
112117

‎codeclash/analysis/metrics/elo.py‎

Lines changed: 45 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ def __init__(
4040
score_type: SCORING_TYPES = "per_round_tertiary",
4141
max_round: int = 15,
4242
only_specific_round: bool = False,
43+
include_round_0: bool = False,
4344
):
4445
"""This class builds a win matrix from a log directory, it doesn't fit anything yet.
4546
It also adds a "ALL" game to the win matrix, which is the sum of all games.
@@ -62,6 +63,9 @@ def __init__(
6263
6364
The `max_round` parameter controls the maximum number of rounds to include in the score calculation (default: 15).
6465
The `only_specific_round` parameter controls whether to only include the specific round (True) or all rounds up to max_round (False).
66+
The `include_round_0` parameter controls whether round 0 is counted. In normal PvP/climbing
67+
tournaments round 0 is the identical-codebases baseline and is excluded. For ladder
68+
construction (`ladder make`, `tournament.rounds: 0`) round 0 IS the match, so set this True.
6569
"""
6670
self.win_matrix: dict[str, dict[tuple[str, str], list[float]]] = defaultdict(
6771
lambda: defaultdict(lambda: [0.0, 0.0])
@@ -71,6 +75,7 @@ def __init__(
7175
self.score_type = score_type
7276
self.max_round = max_round
7377
self.only_specific_round = only_specific_round
78+
self.include_round_0 = include_round_0
7479
self._samples: dict[str, dict[tuple[str, str], list[tuple[float, float]]]] = defaultdict(
7580
lambda: defaultdict(list)
7681
)
@@ -154,13 +159,19 @@ def _process_tournament(self, metadata_path: Path) -> None:
154159
return
155160

156161
player_names = [p["name"] for p in players]
157-
models = [p["config"]["model"]["model_name"].strip("@") for p in players]
162+
models = []
163+
for p in players:
164+
try:
165+
models.append(p["config"]["model"]["model_name"].strip("@"))
166+
except KeyError:
167+
# Ladder bots have no model config; identify by branch (flatten "/" to keep years distinct).
168+
models.append(p["name"].removeprefix("human/").replace("/", "__"))
158169

159170
# Aggregate scores for each round
160171
p1_round_scores = []
161172
p2_round_scores = []
162173
for idx, stats in metadata["round_stats"].items():
163-
if idx == "0":
174+
if idx == "0" and not self.include_round_0:
164175
continue
165176

166177
round_num = int(idx)
@@ -1547,6 +1558,12 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
15471558
parser = argparse.ArgumentParser(description="Build win matrix and fit Bradley-Terry model")
15481559
parser.add_argument("-d", "--log_dir", type=Path, default=LOCAL_LOG_DIR)
15491560
parser.add_argument("--print-matrix", action="store_true", help="Print win matrix")
1561+
parser.add_argument(
1562+
"--include-round-0",
1563+
action="store_true",
1564+
help="Count round 0 (normally the excluded identical-codebases baseline). REQUIRED for "
1565+
"ladder construction (`ladder make` uses tournament.rounds: 0, so round 0 IS the match).",
1566+
)
15501567
parser.add_argument(
15511568
"-s",
15521569
"--score-type",
@@ -1578,7 +1595,9 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
15781595
args = parser.parse_args()
15791596

15801597
builder = ScoreMatrixBuilder(
1581-
all_games_normalization_scheme=args.all_normalization_scheme, score_type=args.score_type
1598+
all_games_normalization_scheme=args.all_normalization_scheme,
1599+
score_type=args.score_type,
1600+
include_round_0=args.include_round_0,
15821601
)
15831602
builder.build(args.log_dir)
15841603

@@ -1632,22 +1651,26 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
16321651
).run()
16331652
write_bootstrap_metrics_table(bootstrap_results, args.output_dir, game="ALL")
16341653

1635-
logger.info("Running EloVsMaxRounds analysis")
1636-
EloVsMaxRounds(
1637-
log_dir=args.log_dir,
1638-
max_rounds=15,
1639-
all_games_normalization_scheme=args.all_normalization_scheme,
1640-
score_type=args.score_type,
1641-
regularization=args.regularization,
1642-
output_dir=args.output_dir,
1643-
).run()
1644-
1645-
logger.info("Running EloOnlyAtRound analysis")
1646-
EloOnlyAtRound(
1647-
log_dir=args.log_dir,
1648-
max_rounds=15,
1649-
all_games_normalization_scheme=args.all_normalization_scheme,
1650-
score_type=args.score_type,
1651-
regularization=args.regularization,
1652-
output_dir=args.output_dir,
1653-
).run()
1654+
# Max-round analyses are multi-round-only; skip them for single-round ladder round-robins.
1655+
if not args.include_round_0:
1656+
logger.info("Running EloVsMaxRounds analysis")
1657+
EloVsMaxRounds(
1658+
log_dir=args.log_dir,
1659+
max_rounds=15,
1660+
all_games_normalization_scheme=args.all_normalization_scheme,
1661+
score_type=args.score_type,
1662+
regularization=args.regularization,
1663+
output_dir=args.output_dir,
1664+
).run()
1665+
1666+
logger.info("Running EloOnlyAtRound analysis")
1667+
EloOnlyAtRound(
1668+
log_dir=args.log_dir,
1669+
max_rounds=15,
1670+
all_games_normalization_scheme=args.all_normalization_scheme,
1671+
score_type=args.score_type,
1672+
regularization=args.regularization,
1673+
output_dir=args.output_dir,
1674+
).run()
1675+
else:
1676+
logger.info("Skipping EloVsMaxRounds / EloOnlyAtRound (ladder mode: single round-0 round-robin)")

‎codeclash/arenas/scml/SCML.Dockerfile‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,13 @@ ENV DEBIAN_FRONTEND=noninteractive \
44
PYTHONDONTWRITEBYTECODE=1 \
55
PIP_NO_CACHE_DIR=1
66

7+
# Pin numpy/BLAS to a single math thread per container. Each SCML simulation is
8+
# single-threaded compute (measured: pinned solo == unpinned solo)
9+
ENV OMP_NUM_THREADS=1 \
10+
OPENBLAS_NUM_THREADS=1 \
11+
MKL_NUM_THREADS=1 \
12+
NUMEXPR_NUM_THREADS=1
13+
714
RUN apt-get update \
815
&& apt-get install -y --no-install-recommends \
916
ca-certificates git build-essential jq \
@@ -12,12 +19,10 @@ RUN apt-get update \
1219
RUN python -m pip install --upgrade pip \
1320
&& python -m pip install scml==0.8.2
1421

22+
# Clone the arena repo so `origin` is set for branch_init / push (matches the other
23+
# arenas). Default branch holds the runtime; human/* branches overlay scml_agent.py.
24+
RUN git clone https://github.com/CodeClash-ai/SCML.git /workspace \
25+
&& cd /workspace \
26+
&& git config user.email "player@codeclash.com" \
27+
&& git config user.name "Player"
1528
WORKDIR /workspace
16-
17-
COPY codeclash/arenas/scml/runtime/ /workspace/
18-
19-
RUN git init \
20-
&& git config user.email "player@codeclash.com" \
21-
&& git config user.name "Player" \
22-
&& git add . \
23-
&& git commit -m "Initial SCML workspace"

‎configs/ladder/make_scml.yaml‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Round-robin over the SCML OneShot human bots (ported from yasserfarouk/scml-agents +
2+
# built-in baselines), used to RANK them via Elo/Bradley-Terry -> see scml.yaml for the ladder.
3+
# rounds:0 = dummy players just play the baseline SCML2024OneShot game (no code edits).
4+
# 51 bots -> 51*50/2 = 1275 pairwise tournaments. Bots live on CodeClash-ai/SCML; Docker required.
5+
#
6+
# COST (measured ~4.8s/sim + ~3s overhead per pair; pairs are single-threaded, so wall =
7+
# core-hours / workers): at sims_per_round=200 a pair takes ~16 min -> ~341 core-hours total.
8+
# * 32-core box (--workers 30): ~11 h * 64 vCPU (-w 62): ~5.5 h * 96 vCPU (-w 90): ~3.8 h
9+
# Keep --workers <= (cores - 2): the decide() call has a 3s timeout that CPU oversubscription
10+
# can trip. Resumable: reruns skip pairs already logged, so it's safe to stop/resume.
11+
#
12+
# Full run (200 sims = publication-quality Elo, sized for SCML's high per-sim variance):
13+
# uv run codeclash ladder make configs/ladder/make_scml.yaml --workers 30
14+
# Cheap SANITY pilot first (~1 h @ 32 cores): temporarily set sims_per_round: 30, run, rank, and
15+
# eyeball the ordering (baselines near the bottom) before committing to the full 200-sim pass.
16+
tournament:
17+
rounds: 0
18+
game:
19+
name: SCML
20+
sims_per_round: 200
21+
timeout: 10800 # 3h per-pair game cap; 200 sims needs ~1-1.5h (default 180s kills the run)
22+
args:
23+
n_steps: 10
24+
n_lines: 2
25+
players:
26+
- agent: dummy
27+
branch_init: human/scml-baselines/greedy
28+
- agent: dummy
29+
branch_init: human/scml-baselines/nice
30+
- agent: dummy
31+
branch_init: human/scml-baselines/random
32+
- agent: dummy
33+
branch_init: human/scml2021/staghunter
34+
- agent: dummy
35+
branch_init: human/scml2021/team_50
36+
- agent: dummy
37+
branch_init: human/scml2021/team_51
38+
- agent: dummy
39+
branch_init: human/scml2021/team_54
40+
- agent: dummy
41+
branch_init: human/scml2021/team_55
42+
- agent: dummy
43+
branch_init: human/scml2021/team_61
44+
- agent: dummy
45+
branch_init: human/scml2021/team_62
46+
- agent: dummy
47+
branch_init: human/scml2021/team_72
48+
- agent: dummy
49+
branch_init: human/scml2021/team_73
50+
- agent: dummy
51+
branch_init: human/scml2021/team_86
52+
- agent: dummy
53+
branch_init: human/scml2021/team_90
54+
- agent: dummy
55+
branch_init: human/scml2021/team_corleone
56+
- agent: dummy
57+
branch_init: human/scml2022/team_102
58+
- agent: dummy
59+
branch_init: human/scml2022/team_103
60+
- agent: dummy
61+
branch_init: human/scml2022/team_105
62+
- agent: dummy
63+
branch_init: human/scml2022/team_106
64+
- agent: dummy
65+
branch_init: human/scml2022/team_107
66+
- agent: dummy
67+
branch_init: human/scml2022/team_123
68+
- agent: dummy
69+
branch_init: human/scml2022/team_124
70+
- agent: dummy
71+
branch_init: human/scml2022/team_126
72+
- agent: dummy
73+
branch_init: human/scml2022/team_131
74+
- agent: dummy
75+
branch_init: human/scml2022/team_134
76+
- agent: dummy
77+
branch_init: human/scml2022/team_94
78+
- agent: dummy
79+
branch_init: human/scml2022/team_96
80+
- agent: dummy
81+
branch_init: human/scml2023/team_102
82+
- agent: dummy
83+
branch_init: human/scml2023/team_123
84+
- agent: dummy
85+
branch_init: human/scml2023/team_126
86+
- agent: dummy
87+
branch_init: human/scml2023/team_127
88+
- agent: dummy
89+
branch_init: human/scml2023/team_134
90+
- agent: dummy
91+
branch_init: human/scml2023/team_143
92+
- agent: dummy
93+
branch_init: human/scml2023/team_144
94+
- agent: dummy
95+
branch_init: human/scml2023/team_145
96+
- agent: dummy
97+
branch_init: human/scml2023/team_148
98+
- agent: dummy
99+
branch_init: human/scml2023/team_149
100+
- agent: dummy
101+
branch_init: human/scml2023/team_151
102+
- agent: dummy
103+
branch_init: human/scml2023/team_poli_usp
104+
- agent: dummy
105+
branch_init: human/scml2024/coyoteteam
106+
- agent: dummy
107+
branch_init: human/scml2024/ozug4
108+
- agent: dummy
109+
branch_init: human/scml2024/team_144
110+
- agent: dummy
111+
branch_init: human/scml2024/team_164
112+
- agent: dummy
113+
branch_init: human/scml2024/team_171
114+
- agent: dummy
115+
branch_init: human/scml2024/team_172
116+
- agent: dummy
117+
branch_init: human/scml2024/team_174
118+
- agent: dummy
119+
branch_init: human/scml2024/team_193
120+
- agent: dummy
121+
branch_init: human/scml2024/team_238
122+
- agent: dummy
123+
branch_init: human/scml2024/team_abc
124+
- agent: dummy
125+
branch_init: human/scml2024/team_miyajima
126+
- agent: dummy
127+
branch_init: human/scml2024/teamyuzuru
128+
prompts:
129+
game_description: SCML OneShot ladder

0 commit comments

Comments
 (0)