Merge branch 'main' of https://github.com/CodeClash-ai/CodeClash into john/gomoku-ladder

# Conflicts:
#	configs/ladder/make_gomoku.yaml

Author: john-b-yang

.claude/skills/create-ladder/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: create-arena-ladder
+description: >-
+  Build a CC:Ladder for any CodeClash arena: import human-written solutions as git
+  branches, rank them via round-robin PvP + Elo, and assemble the ladder configs.
+  Use when asked to "create a ladder", "import human solutions", "push human bots as
+  branches", or "make a CC:<arena> ladder" for arenas like BattleSnake, RobotRumble,
+  CoreWar, Gomoku, RoboCode, SCML, etc.
+---
+
+# Create an Arena Ladder (CC:Ladder)
+
+A **ladder** turns a curated set of human-written bots into a ranked gauntlet, then
+measures how far up a model can climb. Two phases: **make the ladder** (rank the humans
+via round-robin), then **run the ladder** (a model climbs rung by rung until it loses).
+
+This skill produces, for arena `<A>`: `human/<author>/<name>` branches on `CodeClash-ai/<A>`,
+a `make_<a>.yaml` (round-robin), a ranked `rungs/<a>.yaml`, and a run config `<a>.yaml`.
+
+Work in the `repo/` clone. Reference implementations to mirror in
+`configs/ablations/ladder/`: `make_battlesnake.yaml` (round-robin), `battlesnake.yaml`
+(run), `rungs/battlesnake.yaml` (ranked opponents), and its `README.md`. The arena's own
+`codeclash/arenas/<a>/<a>.py` is the source of truth for the submission contract.
+
+---
+
+## Mechanics you must respect (arena-agnostic)
+
+- **Where human code lives:** each arena has its **own** repo under `CodeClash-ai`
+  (hardcoded `GH_ORG` in `codeclash/constants.py`). The arena Dockerfile `git clone`s it
+  into `/workspace`. Human bots are **branches of that per-arena repo**, NOT of this
+  monorepo — so `git branch -a` here shows no `human/*`.
+- **How a branch becomes a player:** a player with `branch_init: human/foo/bar` makes
+  `Player.__init__` (`codeclash/agents/player.py`) run `git fetch && git checkout` in the
+  clone; that branch's files overlay `/workspace`. `agent: dummy` = static opponent.
+  `push: True` (the climbing player) needs `GITHUB_TOKEN`.
+- **What a branch must contain:** the arena's **submission file(s)** at the path its
+  `validate_code` expects; everything else (engine, assets) comes from the base clone.
+  **The single source of truth is `codeclash/arenas/<a>/<a>.py` — read its `submission`
+  attribute and `validate_code` method.** Those two define the contract. Examples:
+  BattleSnake → `main.py` HTTP server (`info/start/end/move`); Gomoku → `main.py` with
+  `get_move(board,color)`; SCML → agent with `decide(observation)`; RoboCode → Java class
+  under `robots/custom/`; CoreWar → `warrior.red`.
+
+---
+
+## Phase 0 — Prerequisites
+
+- Arena class `codeclash/arenas/<a>/<a>.py` exists; record its `submission` path + exact
+  `validate_code` requirements.
+- Arena repo `github.com/CodeClash-ai/<A>` exists and its Dockerfile clones it.
+- Docker running; `GITHUB_TOKEN` set (public repos → `gh auth token` works for push + run).
+
+## Phase 1 — Source many human solutions
+
+The hard part is finding a large set. Best sources: official leaderboards and
+`awesome-<arena>` repos (BattleSnake used `awesome-battlesnake`; RobotRumble crawled
+`robotrumble.org/boards/2`; RoboCode drew from RoboWiki/GitHub). Capture author + bot name
+per candidate → these become the branch slugs. Keep a provenance record (source URL,
+author, license) as you go — a table plus a header comment in each imported file.
+
+## Phase 2 — Adapt each solution to the arena's contract
+
+Every bot must end up matching the one submission contract. Do NOT discard a bot merely for
+being in another language — pick the cheapest import shape:
+
+- **Copy-in** — source is already in the arena's language/framework: drop the files in and
+  rename/repackage (e.g. RoboCode: main class → `MyTank`, `package custom;`). Mechanical.
+- **Function-contract port** — reimplement the core "given state, choose a move" logic as
+  the arena's single entry function (Gomoku `get_move`, SCML `decide`, BattleSnake `move`).
+  Ignore the source's GUI/protocol/CLI wrapper; keep its evaluation + search faithful.
+
+Porting hard rules (adapt per arena; worth writing up as a guide for any porting agents):
+- **Runtime-only deps** — match the arena image (often stdlib-only Python 3.10; re-express
+  array math in pure Python). No trained weights / NN unless you can obtain the binary.
+- **One entry point, never raise** — a crash or illegal move = a forfeit; wrap the body and
+  fall back to a safe legal move.
+- **Fast enough** — a round-robin plays many games; cap search depth / rollouts.
+- Only skip a bot if it truly can't run. **Log every skip with a reason — never silently
+  drop bots.**
+
+## Phase 3 — Validate (two-stage), then push
+
+Local validation is necessary but NOT sufficient — a local shim skips Docker, the repo's
+`server.py`, and real payloads. Gate every bot in two stages before it earns a branch:
+
+1. **Stage 1 (local):** syntax/import + the arena's `validate_code` legality check. Cheap;
+   catches most breakage without Docker. Fix or drop failures.
+2. **Stage 2 (arena, REQUIRED):** play each stage-1 pass through the real arena image and
+   confirm it completes a full game without erroring. This is the step that actually gates.
+   Requires Docker. (Scripting both stages over a folder of candidates is worth it at scale.)
+3. **Push** each stage-2-healthy bot as `human/<author>/<name>` (dedupe identical content).
+   Use consistent kebab/lowercase slugs; branches must be pushed before any arena run, since
+   `branch_init` fetches from the remote.
+
+## Phase 4 — Make the ladder (round-robin + Elo)
+
+1. Write `configs/ablations/ladder/make_<a>.yaml` (mirror `make_battlesnake.yaml`):
+   `tournament.rounds: 0`, a `game` block, and `players:` = every `human/*` branch as
+   `{agent: dummy, branch_init: ...}`.
+2. Pre-build the image **once** to avoid a build stampede under many workers:
+   `docker build -t codeclash/<a> -f codeclash/arenas/<a>/<A>.Dockerfile .`
+3. Run all-pairs PvP (resumable — skips pairs already logged; `--workers ≈ cores-2`):
+   `GITHUB_TOKEN=$(gh auth token) uv run codeclash ladder make configs/ablations/ladder/make_<a>.yaml --workers N`
+   → logs land under `logs/ladder/<A>/`. Fast arenas run on a laptop; big ones (e.g. SCML's
+   ~1275 pairs) want an AWS box under `tmux`/`nohup`.
+4. Rank: `uv run python -m codeclash.analysis.metrics.elo -d logs/ladder/<A> --include-round-0 --output-dir assets/<a>_elo`
+   → prints the Bradley-Terry/Elo order (weakest → strongest); that ordering IS the ladder.
+   - **`--include-round-0` is REQUIRED for ladder construction.** `ladder make` uses
+     `tournament.rounds: 0`, so round 0 IS the match; without the flag every tournament is
+     dropped (round 0 is normally the excluded identical-codebases baseline) and the fit
+     crashes on an empty matrix. Do NOT pass it for normal multi-round PvP/climbing Elo.
+   - Use `uv run python`, not bare `python` — the analysis deps (matplotlib) live in the uv venv.
+   - Tip: run a cheap low-`sims` pilot first and eyeball that baselines sit near the bottom.
+
+## Phase 5 — Assemble the ranked configs
+
+1. `configs/ablations/ladder/rungs/<a>.yaml` — the ranked opponents, **weakest first,
+   strongest last** (each `{agent: dummy, branch_init: human/...}`), in Elo order.
+2. `configs/ablations/ladder/<a>.yaml` — the climber `player` (starting at the weakest
+   rung, `push: True`) + `ladder: !include ablations/ladder/rungs/<a>.yaml` + a
+   `ladder_rules` block. Model on `battlesnake.yaml`.
+3. Optional `<a>__<model>.yaml` per-model variants (swap `model: !include mini/models/...`);
+   they share the same `rungs/<a>.yaml` include.
+
+`ladder_rules` (optional; defaults reproduce historical behavior):
+```yaml
+ladder_rules:
+  min_round_win_fraction: 0.5   # must win strictly more than this fraction of rounds
+  win_last_k: 1                 # ...and must win the last K rounds
+```
+
+## Phase 6 — Run the ladder
+
+`uv run codeclash ladder run configs/ablations/ladder/<a>.yaml`
+→ prints the highest rung reached; logs under `LOCAL_LOG_DIR/<user>/LadderTournament.*`.
+
+---
+
+## Deliverables checklist
+- [ ] N validated `human/<author>/<name>` branches pushed to `CodeClash-ai/<A>`.
+- [ ] **Stage-2 arena smoke passed** (each bot plays a real game in Docker), skips logged.
+- [ ] `make_<a>.yaml` + round-robin run + Elo ranking (`assets/<a>_elo`).
+- [ ] `rungs/<a>.yaml` (weakest→strongest) + `<a>.yaml` (climber + `ladder_rules`).
+- [ ] `ladder run` executes end-to-end against a sample model.
+- [ ] Provenance recorded (source/author/license per bot); note in `ladder/README.md` if
+      the arena is new.
+
+## Gotchas
+- Human branches go to the **per-arena** repo, not the monorepo; `branch_init` fetches from
+  the remote, so **push before any run** and keep Docker up.
+- A bot that fails `validate_code` silently forfeits. A local shim can pass yet fail in the
+  arena — the **stage-2 Docker smoke** is the real gate, not stage 1.
+- Pre-build the arena image once before a `--workers N` run, or workers stampede the build.
+- Port aggressively into the single submission contract; skip only un-runnable bots, and
+  log every skip. A port must reproduce the original's behavior.

README.md

@@ -73,7 +73,7 @@ codeclash run configs/test/battlesnake.yaml
 Once this works, you should be set up to run a real tournament!
 To run *Claude Sonnet 4.5* against *o3* in a *BattleSnake* tournament with *5 rounds* and *1000 competition simulations* per round, run:
 ```bash
-uv run codeclash run configs/examples/BattleSnake__claude-sonnet-4-5-20250929__o3__r5__s1000.yaml
+uv run codeclash run configs/pvp/BattleSnake__claude-sonnet-4-5-20250929__o3__r15__s1000.yaml
 ```
 
 ## ⚔️ How It Works

codeclash/agents/mini_anthropic_model.py

@@ -11,7 +11,7 @@
 `model_class: codeclash.agents.mini_anthropic_model.AnthropicModel` and provide the API key,
 base URL, and model name (see the `*_env` config fields, which keep endpoint-specific values in
 the environment rather than in committed configs). Requires the optional `anthropic` dependency
-(`uv pip install -e '.[llama]'`). See configs/ablations/ladder/robotrumble_llama.yaml.
+(`uv pip install -e '.[llama]'`). See configs/ladder/robotrumble_llama.yaml.
 """
 
 import json

codeclash/agents/utils.py

@@ -26,6 +26,7 @@ class GameContext(BaseModel):
     round: int
     rounds: int
     working_dir: str
+    arena_description: str = ""
 
     def _render_prompt_templates(self) -> dict:
         context = self.model_dump()

codeclash/analysis/code_evolve/main.py

@@ -18,7 +18,7 @@
 from codeclash.arenas import ARENAS
 from codeclash.constants import LOCAL_LOG_DIR
 
-MODELS_PATH = Path("configs/models.yaml")
+MODELS_PATH = Path("configs/mini/model_roster.yaml")
 TARGET_ROUNDS = [1, 15, 5, 10]
 
 

codeclash/analysis/metrics/elo.py

@@ -40,6 +40,7 @@ def __init__(
         score_type: SCORING_TYPES = "per_round_tertiary",
         max_round: int = 15,
         only_specific_round: bool = False,
+        include_round_0: bool = False,
     ):
         """This class builds a win matrix from a log directory, it doesn't fit anything yet.
         It also adds a "ALL" game to the win matrix, which is the sum of all games.
@@ -62,6 +63,9 @@ def __init__(
 
         The `max_round` parameter controls the maximum number of rounds to include in the score calculation (default: 15).
         The `only_specific_round` parameter controls whether to only include the specific round (True) or all rounds up to max_round (False).
+        The `include_round_0` parameter controls whether round 0 is counted. In normal PvP/climbing
+        tournaments round 0 is the identical-codebases baseline and is excluded. For ladder
+        construction (`ladder make`, `tournament.rounds: 0`) round 0 IS the match, so set this True.
         """
         self.win_matrix: dict[str, dict[tuple[str, str], list[float]]] = defaultdict(
             lambda: defaultdict(lambda: [0.0, 0.0])
@@ -71,6 +75,7 @@ def __init__(
         self.score_type = score_type
         self.max_round = max_round
         self.only_specific_round = only_specific_round
+        self.include_round_0 = include_round_0
         self._samples: dict[str, dict[tuple[str, str], list[tuple[float, float]]]] = defaultdict(
             lambda: defaultdict(list)
         )
@@ -154,13 +159,19 @@ def _process_tournament(self, metadata_path: Path) -> None:
             return
 
         player_names = [p["name"] for p in players]
-        models = [p["config"]["model"]["model_name"].strip("@") for p in players]
+        models = []
+        for p in players:
+            try:
+                models.append(p["config"]["model"]["model_name"].strip("@"))
+            except KeyError:
+                # Ladder bots have no model config; identify by branch (flatten "/" to keep years distinct).
+                models.append(p["name"].removeprefix("human/").replace("/", "__"))
 
         # Aggregate scores for each round
         p1_round_scores = []
         p2_round_scores = []
         for idx, stats in metadata["round_stats"].items():
-            if idx == "0":
+            if idx == "0" and not self.include_round_0:
                 continue
 
             round_num = int(idx)
@@ -1547,6 +1558,12 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
     parser = argparse.ArgumentParser(description="Build win matrix and fit Bradley-Terry model")
     parser.add_argument("-d", "--log_dir", type=Path, default=LOCAL_LOG_DIR)
     parser.add_argument("--print-matrix", action="store_true", help="Print win matrix")
+    parser.add_argument(
+        "--include-round-0",
+        action="store_true",
+        help="Count round 0 (normally the excluded identical-codebases baseline). REQUIRED for "
+        "ladder construction (`ladder make` uses tournament.rounds: 0, so round 0 IS the match).",
+    )
     parser.add_argument(
         "-s",
         "--score-type",
@@ -1578,7 +1595,9 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
     args = parser.parse_args()
 
     builder = ScoreMatrixBuilder(
-        all_games_normalization_scheme=args.all_normalization_scheme, score_type=args.score_type
+        all_games_normalization_scheme=args.all_normalization_scheme,
+        score_type=args.score_type,
+        include_round_0=args.include_round_0,
     )
     builder.build(args.log_dir)
 
@@ -1632,22 +1651,26 @@ def write_latex_table_plain(results: dict[str, dict], output_dir: Path) -> None:
             ).run()
         write_bootstrap_metrics_table(bootstrap_results, args.output_dir, game="ALL")
 
-    logger.info("Running EloVsMaxRounds analysis")
-    EloVsMaxRounds(
-        log_dir=args.log_dir,
-        max_rounds=15,
-        all_games_normalization_scheme=args.all_normalization_scheme,
-        score_type=args.score_type,
-        regularization=args.regularization,
-        output_dir=args.output_dir,
-    ).run()
-
-    logger.info("Running EloOnlyAtRound analysis")
-    EloOnlyAtRound(
-        log_dir=args.log_dir,
-        max_rounds=15,
-        all_games_normalization_scheme=args.all_normalization_scheme,
-        score_type=args.score_type,
-        regularization=args.regularization,
-        output_dir=args.output_dir,
-    ).run()
+    # Max-round analyses are multi-round-only; skip them for single-round ladder round-robins.
+    if not args.include_round_0:
+        logger.info("Running EloVsMaxRounds analysis")
+        EloVsMaxRounds(
+            log_dir=args.log_dir,
+            max_rounds=15,
+            all_games_normalization_scheme=args.all_normalization_scheme,
+            score_type=args.score_type,
+            regularization=args.regularization,
+            output_dir=args.output_dir,
+        ).run()
+
+        logger.info("Running EloOnlyAtRound analysis")
+        EloOnlyAtRound(
+            log_dir=args.log_dir,
+            max_rounds=15,
+            all_games_normalization_scheme=args.all_normalization_scheme,
+            score_type=args.score_type,
+            regularization=args.regularization,
+            output_dir=args.output_dir,
+        ).run()
+    else:
+        logger.info("Skipping EloVsMaxRounds / EloOnlyAtRound (ladder mode: single round-0 round-robin)")

codeclash/arenas/arena.py

@@ -2,6 +2,7 @@
 import os
 import random
 import subprocess
+import threading
 import time
 from abc import ABC, abstractmethod
 from pathlib import Path
@@ -75,6 +76,10 @@ class CodeArena(ABC):
     default_args: dict = {}
     submission: str
 
+    # Serializes image builds across concurrent pairs (e.g. `ladder make --workers N`), so
+    # worker threads don't all race `docker build` on a cold start and fail with "already exists".
+    _build_lock = threading.Lock()
+
     def __init__(self, config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False):
         """The CodeArena class is responsible for running games, i.e., taking a list of code
         from different agents/players and running them against each other.
@@ -127,36 +132,38 @@ def build_image(self):
         if is_running_in_aws_batch():
             pull_game_container_aws_ecr(game_name=self.name, image_name=self.image_name, logger=self.logger)
 
-        # Check if container exists using subprocess
-        self.logger.debug(f"Checking if container {self.image_name} exists")
-        result = subprocess.run(
-            f"docker images -q {self.image_name}",
-            shell=True,
-            capture_output=True,
-            text=True,
-        )
-        if result.stdout.strip():
-            self.logger.debug(f"Container {self.image_name} exists")
-            return
+        # Hold the lock across check-and-build so concurrent pairs don't race: the first thread
+        # builds while the rest wait, then find the image already present and skip.
+        with CodeArena._build_lock:
+            self.logger.debug(f"Checking if container {self.image_name} exists")
+            result = subprocess.run(
+                f"docker images -q {self.image_name}",
+                shell=True,
+                capture_output=True,
+                text=True,
+            )
+            if result.stdout.strip():
+                self.logger.debug(f"Container {self.image_name} exists")
+                return
 
-        self.logger.info(
-            f"Building Docker image {self.image_name}. This may take 1-5 minutes and only work on Linux for some games."
-        )
+            self.logger.info(
+                f"Building Docker image {self.image_name}. This may take 1-5 minutes and only work on Linux for some games."
+            )
 
-        # NOTE: Assuming Dockerfile is declared in same directory as the arena.
-        arena_file = Path(inspect.getfile(self.__class__))
-        folder_path = arena_file.parent
-        result = subprocess.run(
-            f"docker build --no-cache -t {self.image_name} -f {folder_path}/{self.name}.Dockerfile .",
-            shell=True,
-            capture_output=True,
-            text=True,
-        )
-        if result.returncode == 0:
-            self.logger.info(f"✅ Built Docker image {self.image_name}")
-        else:
-            self.logger.error(f"❌ Failed to build Docker image: {result.stderr}\n{result.stdout}{result.stderr}")
-            raise RuntimeError(f"Failed to build Docker image: {result.stderr}")
+            # NOTE: Assuming Dockerfile is declared in same directory as the arena.
+            arena_file = Path(inspect.getfile(self.__class__))
+            folder_path = arena_file.parent
+            result = subprocess.run(
+                f"docker build --no-cache -t {self.image_name} -f {folder_path}/{self.name}.Dockerfile .",
+                shell=True,
+                capture_output=True,
+                text=True,
+            )
+            if result.returncode == 0:
+                self.logger.info(f"✅ Built Docker image {self.image_name}")
+            else:
+                self.logger.error(f"❌ Failed to build Docker image: {result.stderr}\n{result.stdout}{result.stderr}")
+                raise RuntimeError(f"Failed to build Docker image: {result.stderr}")
 
     def copy_logs_from_env(self, round_num: int) -> None:
         """Copy logs from the game's environment to the local machine."""