Clean up ladder configs, agent observations, and BattleSnake validation (#116)

* Add lock for image construction

* Update win conditions

* Surface per-step budget in agent observations

Add a global observation_template in configs/mini/default.yaml with a
[Step n/limit used - k left] banner, and thread it onto the model via a
ClashAgentConfig field pushed in ClashAgent.__init__. Applies to every model
backend, tournament type, and arena; models otherwise learn the step limit only
once at round start with no running counter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Reject BattleSnake submissions missing a server entrypoint

validate_code was a pure function-def check, so a main.py that drops the
'if __name__ == "__main__": run_server(...)' block passed validation but
failed to start at runtime (forfeit). Require the entrypoint statically.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Centralize ladder win condition in shared ladder_rules.yaml

Every ladder run config duplicated the same ladder_rules block. Extract it to
configs/ablations/ladder/ladder_rules.yaml (min_round_wins: 2, win_last_k: 0)
and include it from all run configs so the win condition lives in one place.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add matplotlib to dev dependencies

Required by codeclash.analysis (elo plots + ranking).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Print ladder advancement rule to console

Unify with scml-ladder's behavior: echo the advancement rule to stdout (in
addition to logging it) at the start of a ladder run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* reduce ocmments

* nit

* reduce comments

* Give ladder climbers an adversarial task prompt

Ladder run configs only set game_description to e.g. 'BattleSnake ladder', so a
climbing model was never told it's writing code to defeat an opponent (the shared
instance template even frames rounds as a cooperative relay). Add a shared
ladder_prompt.yaml with an arena-agnostic prompt that states the adversarial task
and pulls in arena specifics via a new {{arena_description}} template var (exposed
from the arena's static description through GameContext). All 19 run configs
include it; make_* build configs are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Strip explanatory comments from ladder/agent config additions

* Fix pre-commit

* Move observation_template from agent config into model configs

* Reorganize configs: main->pvp, ladder out of ablations, models.yaml->mini/model_roster.yaml

* Remove unreferenced example configs (stale Bridge/haiku, duplicate HuskyBench)

* Point docs at pvp configs; drop r5 example duplicates and vs_human ablation

* Fix battlesnake entrypoint check to accept any server start; fix single_player doc ref

* Remove unused ClashAgentConfig now that observation_template lives in model configs

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: john-b-yang

README.md

@@ -73,7 +73,7 @@ codeclash run configs/test/battlesnake.yaml
 Once this works, you should be set up to run a real tournament!
 To run *Claude Sonnet 4.5* against *o3* in a *BattleSnake* tournament with *5 rounds* and *1000 competition simulations* per round, run:
 ```bash
-uv run codeclash run configs/examples/BattleSnake__claude-sonnet-4-5-20250929__o3__r5__s1000.yaml
+uv run codeclash run configs/pvp/BattleSnake__claude-sonnet-4-5-20250929__o3__r15__s1000.yaml
 ```
 
 ## ⚔️ How It Works

codeclash/agents/mini_anthropic_model.py

@@ -11,7 +11,7 @@
 `model_class: codeclash.agents.mini_anthropic_model.AnthropicModel` and provide the API key,
 base URL, and model name (see the `*_env` config fields, which keep endpoint-specific values in
 the environment rather than in committed configs). Requires the optional `anthropic` dependency
-(`uv pip install -e '.[llama]'`). See configs/ablations/ladder/robotrumble_llama.yaml.
+(`uv pip install -e '.[llama]'`). See configs/ladder/robotrumble_llama.yaml.
 """
 
 import json

codeclash/agents/utils.py

@@ -26,6 +26,7 @@ class GameContext(BaseModel):
     round: int
     rounds: int
     working_dir: str
+    arena_description: str = ""
 
     def _render_prompt_templates(self) -> dict:
         context = self.model_dump()

codeclash/analysis/code_evolve/main.py

@@ -18,7 +18,7 @@
 from codeclash.arenas import ARENAS
 from codeclash.constants import LOCAL_LOG_DIR
 
-MODELS_PATH = Path("configs/models.yaml")
+MODELS_PATH = Path("configs/mini/model_roster.yaml")
 TARGET_ROUNDS = [1, 15, 5, 10]
 
 

codeclash/arenas/arena.py

@@ -2,6 +2,7 @@
 import os
 import random
 import subprocess
+import threading
 import time
 from abc import ABC, abstractmethod
 from pathlib import Path
@@ -75,6 +76,10 @@ class CodeArena(ABC):
     default_args: dict = {}
     submission: str
 
+    # Serializes image builds across concurrent pairs (e.g. `ladder make --workers N`), so
+    # worker threads don't all race `docker build` on a cold start and fail with "already exists".
+    _build_lock = threading.Lock()
+
     def __init__(self, config: dict, *, tournament_id: str, local_output_dir: Path, keep_containers: bool = False):
         """The CodeArena class is responsible for running games, i.e., taking a list of code
         from different agents/players and running them against each other.
@@ -127,36 +132,38 @@ def build_image(self):
         if is_running_in_aws_batch():
             pull_game_container_aws_ecr(game_name=self.name, image_name=self.image_name, logger=self.logger)
 
-        # Check if container exists using subprocess
-        self.logger.debug(f"Checking if container {self.image_name} exists")
-        result = subprocess.run(
-            f"docker images -q {self.image_name}",
-            shell=True,
-            capture_output=True,
-            text=True,
-        )
-        if result.stdout.strip():
-            self.logger.debug(f"Container {self.image_name} exists")
-            return
+        # Hold the lock across check-and-build so concurrent pairs don't race: the first thread
+        # builds while the rest wait, then find the image already present and skip.
+        with CodeArena._build_lock:
+            self.logger.debug(f"Checking if container {self.image_name} exists")
+            result = subprocess.run(
+                f"docker images -q {self.image_name}",
+                shell=True,
+                capture_output=True,
+                text=True,
+            )
+            if result.stdout.strip():
+                self.logger.debug(f"Container {self.image_name} exists")
+                return
 
-        self.logger.info(
-            f"Building Docker image {self.image_name}. This may take 1-5 minutes and only work on Linux for some games."
-        )
+            self.logger.info(
+                f"Building Docker image {self.image_name}. This may take 1-5 minutes and only work on Linux for some games."
+            )
 
-        # NOTE: Assuming Dockerfile is declared in same directory as the arena.
-        arena_file = Path(inspect.getfile(self.__class__))
-        folder_path = arena_file.parent
-        result = subprocess.run(
-            f"docker build --no-cache -t {self.image_name} -f {folder_path}/{self.name}.Dockerfile .",
-            shell=True,
-            capture_output=True,
-            text=True,
-        )
-        if result.returncode == 0:
-            self.logger.info(f"✅ Built Docker image {self.image_name}")
-        else:
-            self.logger.error(f"❌ Failed to build Docker image: {result.stderr}\n{result.stdout}{result.stderr}")
-            raise RuntimeError(f"Failed to build Docker image: {result.stderr}")
+            # NOTE: Assuming Dockerfile is declared in same directory as the arena.
+            arena_file = Path(inspect.getfile(self.__class__))
+            folder_path = arena_file.parent
+            result = subprocess.run(
+                f"docker build --no-cache -t {self.image_name} -f {folder_path}/{self.name}.Dockerfile .",
+                shell=True,
+                capture_output=True,
+                text=True,
+            )
+            if result.returncode == 0:
+                self.logger.info(f"✅ Built Docker image {self.image_name}")
+            else:
+                self.logger.error(f"❌ Failed to build Docker image: {result.stderr}\n{result.stdout}{result.stderr}")
+                raise RuntimeError(f"Failed to build Docker image: {result.stderr}")
 
     def copy_logs_from_env(self, round_num: int) -> None:
         """Copy logs from the game's environment to the local machine."""

codeclash/arenas/battlesnake/battlesnake.py

@@ -203,4 +203,9 @@ def validate_code(self, agent: Player) -> tuple[bool, str | None]:
                 error_msg.append(f"There should be a `{func}` function implemented in `{self.submission}`")
         if len(error_msg) > 0:
             return False, "\n".join(error_msg + ["Don't change the function signatures!"])
+        if "__main__" not in bot_content:
+            return False, (
+                f'`{self.submission}` must keep its `if __name__ == "__main__"` block that starts '
+                "the server, or the bot fails to launch."
+            )
         return True, None

codeclash/cli/ladder.py

@@ -2,6 +2,7 @@
 
 import copy
 import getpass
+import json
 import time
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from pathlib import Path
@@ -18,43 +19,53 @@
 logger = get_logger("ladder")
 
 
-def _resolve_ladder_rules(ladder_rules: dict, rounds: int) -> tuple[float, int]:
-    """Validate the optional ``ladder_rules`` block and return ``(min_round_win_fraction, win_last_k)``.
+def _resolve_ladder_rules(ladder_rules: dict, rounds: int) -> tuple[int, int]:
+    """Validate the required ``ladder_rules`` block and return ``(min_round_wins, win_last_k)``.
 
-    Defaults: win at least ``min_round_win_fraction`` (0.4) of the *agent* rounds AND win the last
-    ``win_last_k`` (1) round(s). The baseline round 0 (identical, un-edited codebases) is excluded
-    from this count — it reflects game variance, not the agent — so the fraction is taken over the
-    ``rounds`` rounds the agent actually edits.
+    Both keys must be specified explicitly in the config (no defaults):
+    - ``min_round_wins``: the whole number of *agent* rounds the player must win to advance
+      (a ``>=`` threshold). Must be ``1 <= min_round_wins <= rounds``.
+    - ``win_last_k``: the player must win the last ``win_last_k`` round(s). ``1`` means just the final
+      round; ``0`` disables the trailing-rounds requirement entirely. Must be ``<= min_round_wins``.
+
+    The baseline round 0 (identical, un-edited codebases) is excluded from the count — it reflects
+    game variance, not the agent — so wins are counted over the ``rounds`` rounds the agent actually
+    edits (rounds 1..``rounds``).
     """
-    min_round_win_fraction = ladder_rules.get("min_round_win_fraction", 0.4)
-    win_last_k = ladder_rules.get("win_last_k", 1)
+    if "min_round_wins" not in ladder_rules:
+        typer.echo("ladder_rules.min_round_wins is required; specify it explicitly in the config.")
+        raise typer.Exit(1)
+    if "win_last_k" not in ladder_rules:
+        typer.echo("ladder_rules.win_last_k is required; specify it explicitly in the config.")
+        raise typer.Exit(1)
+    min_round_wins = ladder_rules["min_round_wins"]
+    win_last_k = ladder_rules["win_last_k"]
+
+    # min_round_wins: whole number of agent rounds the player must win (round 0 excluded).
+    if isinstance(min_round_wins, bool) or not isinstance(min_round_wins, int):
+        typer.echo(f"ladder_rules.min_round_wins must be an integer, got {min_round_wins!r}.")
+        raise typer.Exit(1)
+    if not 1 <= min_round_wins <= rounds:
+        typer.echo(f"ladder_rules.min_round_wins must be in [1, {rounds}] (tournament.rounds), got {min_round_wins}.")
+        raise typer.Exit(1)
 
-    # win_last_k: number of trailing rounds the player must win (1 == just the final round).
+    # win_last_k: number of trailing rounds the player must win (1 == just the final round, 0 == disabled).
     if isinstance(win_last_k, bool) or not isinstance(win_last_k, int):
         typer.echo(f"ladder_rules.win_last_k must be an integer, got {win_last_k!r}.")
         raise typer.Exit(1)
-    if win_last_k < 1:
+    if win_last_k < 0:
         typer.echo(
-            f"ladder_rules.win_last_k must be >= 1, got {win_last_k}. Use 1 to require winning only the final round."
+            f"ladder_rules.win_last_k must be >= 0, got {win_last_k}. "
+            "Use 0 to disable the trailing-rounds requirement, or 1 to require winning only the final round."
         )
         raise typer.Exit(1)
-    if win_last_k > rounds:
-        typer.echo(f"ladder_rules.win_last_k ({win_last_k}) cannot exceed tournament.rounds ({rounds}).")
-        raise typer.Exit(1)
-
-    # min_round_win_fraction: player must win >= this fraction of the agent rounds (round 0 excluded).
-    if isinstance(min_round_win_fraction, bool) or not isinstance(min_round_win_fraction, (int, float)):
-        typer.echo(f"ladder_rules.min_round_win_fraction must be a number, got {min_round_win_fraction!r}.")
-        raise typer.Exit(1)
-    if not 0 <= min_round_win_fraction <= 1:
+    if win_last_k > min_round_wins:
         typer.echo(
-            f"ladder_rules.min_round_win_fraction must be in [0, 1], got {min_round_win_fraction}. "
-            "The player must win >= this fraction of the agent rounds; 1 requires winning all of them, "
-            "0 drops the fraction requirement."
+            f"ladder_rules.win_last_k ({win_last_k}) cannot exceed ladder_rules.min_round_wins ({min_round_wins})."
         )
         raise typer.Exit(1)
 
-    return float(min_round_win_fraction), win_last_k
+    return min_round_wins, win_last_k
 
 
 ladder_app = typer.Typer(
@@ -74,7 +85,7 @@ def make(
 ):
     """Build a ladder: run PvP tournaments across all pairs of players (for ranking).
 
-    [dim]• codeclash ladder make configs/ablations/ladder/make_battlesnake.yaml[/dim]
+    [dim]• codeclash ladder make configs/ladder/make_battlesnake.yaml[/dim]
     """
     yaml_content = config_path.read_text()
     preprocessed_yaml = resolve_includes(yaml_content, base_dir=CONFIG_DIR)
@@ -141,20 +152,25 @@ def run(
         config["tournament"]["rounds"],
         config["game"]["sims_per_round"],
     )
-    min_round_win_fraction, win_last_k = _resolve_ladder_rules(config.get("ladder_rules", {}), rounds)
+    min_round_wins, win_last_k = _resolve_ladder_rules(config.get("ladder_rules", {}), rounds)
     timestamp = time.strftime("%y%m%d%H%M%S")
     del config["player"]
     del config["ladder"]
     config.pop("ladder_rules", None)
 
-    print(
-        f"Ladder advancement rule: win >= {min_round_win_fraction:.0%} of {rounds} agent rounds "
-        f"(baseline round 0 excluded) and win the last {win_last_k} round(s)."
+    last_k_rule = "disabled" if win_last_k == 0 else f"win the last {win_last_k} round(s)"
+    advancement_rule = (
+        f"Ladder advancement rule: win >= {min_round_wins} of {rounds} agent rounds "
+        f"(baseline round 0 excluded) and {last_k_rule}."
     )
+    print(advancement_rule)
+    logger.info(advancement_rule)
     ladder_folder = f"LadderTournament.{config['game']['name']}.r{rounds}.s{sims}.{player['name']}.{timestamp}"
     player["branch"] = ladder_folder
     parent_dir = LOCAL_LOG_DIR / getpass.getuser() / ladder_folder
 
+    rungs_cleared = 0
+    advanced = False
     for idx, opponent in enumerate(ladder):
         opponent_rank = len(ladder) - idx
         opponent["name"] = opponent["branch_init"].replace("human/", "").replace("/", "_")
@@ -190,24 +206,37 @@ def run(
             metadata = yaml.safe_load(f)
         round_winners = [r["winner"] for k, r in metadata["round_stats"].items() if int(k) != 0]
 
-        # Advancement rule (configurable via `ladder_rules`): win at least
-        # `min_round_win_fraction` of the agent rounds AND win the last `win_last_k` rounds.
+        # Advancement rule (required via `ladder_rules`): win at least `min_round_wins` of the
+        # agent rounds AND win the last `win_last_k` rounds. win_last_k == 0 disables the
+        # trailing-rounds requirement.
         player_wins = sum(1 for w in round_winners if w == player["name"])
-        won_majority = player_wins >= len(round_winners) * min_round_win_fraction
-        won_last_k = all(w == player["name"] for w in round_winners[-win_last_k:])
+        won_majority = player_wins >= min_round_wins
+        won_last_k = win_last_k == 0 or all(w == player["name"] for w in round_winners[-win_last_k:])
+        advanced = won_majority and won_last_k
 
-        if not won_majority or not won_last_k:
+        # Record this rung's outcome in its metadata.json (durable gameplay log). The rule itself
+        # (min_round_wins, win_last_k) is constant across the run and lives in the ladder summary.
+        metadata["ladder_advancement"] = {
+            "player_wins": player_wins,
+            "won_last_k": won_last_k,
+            "cleared": advanced,
+        }
+        with open(metadata_path, "w") as f:
+            json.dump(metadata, f, indent=2)
+
+        if not advanced:
             # Player failed the advancement rule; the ladder challenge ends here.
             print("=" * 10)
             print(
                 f"{player['name']} did not clear {opponent['name']} "
                 f"(rank {opponent_rank}/{len(ladder)}): won {player_wins}/{len(round_winners)} agent rounds "
-                f"(needed >= {min_round_win_fraction:.0%}), last {win_last_k} round(s) won: {won_last_k}.\n"
+                f"(needed >= {min_round_wins}), last {win_last_k} round(s) won: {won_last_k}.\n"
                 "Ladder challenge ends."
             )
             print("=" * 10)
             break
 
+        rungs_cleared += 1
         print("=" * 10)
         print(
             f"{player['name']} successfully beat {opponent['name']} (rank {opponent_rank}/{len(ladder)}) "
@@ -216,5 +245,22 @@ def run(
         )
         print("=" * 10)
 
+    # Persist the overall climb result to a ladder-level metadata.json in the run's parent dir.
+    ladder_summary = {
+        "player": player["name"],
+        "game": config["game"]["name"],
+        "rounds": rounds,
+        "min_round_wins": min_round_wins,
+        "win_last_k": win_last_k,
+        "ladder_size": len(ladder),
+        "rungs_cleared": rungs_cleared,
+        "final_opponent": opponent["name"],
+        "final_opponent_rank": opponent_rank,
+        "cleared_ladder": rungs_cleared == len(ladder),
+    }
+    parent_dir.mkdir(parents=True, exist_ok=True)
+    with open(parent_dir / "metadata.json", "w") as f:
+        json.dump(ladder_summary, f, indent=2)
+
     print(f"Ladder tournament complete. Logs saved to {parent_dir}")
     print(f"Final opponent faced: {opponent['name']} (rank {opponent_rank}/{len(ladder)} in ladder)")

codeclash/tournaments/pvp.py

@@ -79,6 +79,7 @@ def get_agent(self, agent_config: dict, prompts: dict) -> Player:
             round=1,
             rounds=self.rounds,
             working_dir=str(DIR_WORK),
+            arena_description=self.game.description,
         )
 
         return get_agent(agent_config, game_context, environment)

codeclash/utils/generate_confs.py

@@ -3,14 +3,14 @@
 
 Each configuration file specifies a tournament between two models in a given arena,
 including the number of rounds and simulations per round. The configurations are saved
-as YAML files in the specified output directory (default: configs/main/).
+as YAML files in the specified output directory (default: configs/pvp/).
 
 Also generates a tracking JSON file at configs/tracker.json to keep track of
 the number of tournaments and rounds played for each pair of models in each arena.
 
 Usage:
 
-python codeclash/utils/generate_confs.py -m configs/models.yaml -r 15 -s 1000
+python codeclash/utils/generate_confs.py -m configs/mini/model_roster.yaml -r 15 -s 1000
 """
 
 import argparse
@@ -184,7 +184,7 @@ def main(models, arenas, rounds: int, simulations: int, record_ratio: float, out
         "-m",
         "--models",
         type=str,
-        default="configs/models.yaml",
+        default="configs/mini/model_roster.yaml",
         help="Path to model configurations.",
     )
     parser.add_argument(
@@ -218,7 +218,7 @@ def main(models, arenas, rounds: int, simulations: int, record_ratio: float, out
         "-o",
         "--output",
         type=Path,
-        default=Path("configs/main/"),
+        default=Path("configs/pvp/"),
         help="Output directory for configuration files (default: main/).",
     )
     args = parser.parse_args()

codeclash/viewer/static/js/picker.js

@@ -503,7 +503,7 @@ async function fillTextareaWithAWSSubmitCommands() {
     // Format output as AWS submit commands
     const commands = successfulResults.map(
       (result) =>
-        `aws/run_job.py -- aws/docker_and_sync.sh codeclash run configs/main/${result.config_name}`,
+        `aws/run_job.py -- aws/docker_and_sync.sh codeclash run configs/pvp/${result.config_name}`,
     );
     textarea.value = commands.join("\n");