Skip to content

Commit baa1c87

Browse files
committed
Add SCML OneShot ladder: wire arena repo + import 51 human bots
- Create/wire SCML for the ladder: SCML.Dockerfile now git-clones CodeClash-ai/SCML (seeded with the runtime) so branch_init works, matching the other arenas. - Import 51 human bots (3 built-in baselines + 48 ANAC agents 2021-2024 ported from yasserfarouk/scml-agents) to human/* branches, each validated in the real runtime. - make_scml.yaml round-robin (400 sims) for Elo ranking; scml_sources.md manifest; scml_pilot/ tooling (porting guide, validators, push script, AWS runbook); game viz. - team_139 deferred (extreme-price bound mismatch).
1 parent c16a9c2 commit baa1c87

68 files changed

Lines changed: 8372 additions & 8 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

codeclash/arenas/scml/SCML.Dockerfile

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,11 @@ RUN apt-get update \
1212
RUN python -m pip install --upgrade pip \
1313
&& python -m pip install scml==0.8.2
1414

15+
# Clone the arena repo so `origin` is set for branch_init / push (matches the other
16+
# arenas). Default branch holds the runtime; human/* branches overlay scml_agent.py.
17+
RUN git clone https://github.com/CodeClash-ai/SCML.git /workspace \
18+
&& cd /workspace \
19+
&& git remote set-url origin https://github.com/CodeClash-ai/SCML.git \
20+
&& git config user.email "player@codeclash.com" \
21+
&& git config user.name "Player"
1522
WORKDIR /workspace
16-
17-
COPY codeclash/arenas/scml/runtime/ /workspace/
18-
19-
RUN git init \
20-
&& git config user.email "player@codeclash.com" \
21-
&& git config user.name "Player" \
22-
&& git add . \
23-
&& git commit -m "Initial SCML workspace"

configs/ablations/ladder/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,18 @@ You can follow these steps to create your own "CC:<arena>" ladder.
1212
The tricky part is typically finding a large collection of human solutions for a particular arena.
1313
We've typically found that googling for online leaderboards or awesome-<arena> repositories (e.g. [BattleSnake](https://github.com/BattlesnakeOfficial/awesome-battlesnake)) is a good strategy.
1414

15+
### SCML OneShot (newly added)
16+
17+
The [CC:SCML](https://github.com/CodeClash-ai/SCML) repo hosts 51 human bots on `human/*` branches:
18+
3 built-in `scml` baselines (greedy/random/nice) plus 48 ANAC competition agents ported from
19+
[`yasserfarouk/scml-agents`](https://github.com/yasserfarouk/scml-agents) (years 2021–2024). Each
20+
was re-expressed from its `OneShotAgent`/`OneShotSyncAgent` source into the arena's single-file
21+
`decide(observation)` contract (see `scml_pilot/PORTING_GUIDE.md` and `scml_sources.md`). Candidate
22+
sourcing, the porting recipe, and the validation/push tooling live in `scml_pilot/`. Note: unlike
23+
the earlier arenas, SCML's `git`-workspace had to be wired for the ladder — the `CodeClash-ai/SCML`
24+
repo was created (seeded with the runtime) and `SCML.Dockerfile` now `git clone`s it so `branch_init`
25+
works.
26+
1527
## Config layout
1628

1729
Each arena has a few kinds of config in this folder:
Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Round-robin over the SCML OneShot human bots (ported from yasserfarouk/scml-agents +
2+
# built-in baselines), used to RANK them via Elo/Bradley-Terry -> see scml.yaml for the ladder.
3+
# rounds:0 = dummy players just play the baseline SCML2024OneShot game (no code edits).
4+
# 51 bots -> 51*50/2 = 1275 pairwise tournaments. Bots live on CodeClash-ai/SCML; Docker required.
5+
#
6+
# COST (measured ~4.8s/sim + ~3s overhead per pair; pairs are single-threaded, so wall =
7+
# core-hours / workers): at sims_per_round=400 a pair takes ~32 min -> ~681 core-hours total.
8+
# * 32-core box (--workers 30): ~23 h * 64 vCPU (-w 62): ~11 h * 96 vCPU (-w 90): ~7.5 h
9+
# Keep --workers <= (cores - 2): the decide() call has a 3s timeout that CPU oversubscription
10+
# can trip. Resumable: reruns skip pairs already logged, so it's safe to stop/resume.
11+
#
12+
# Full run (250-400 sims = publication-quality Elo; 400 chosen for SCML's high per-sim variance):
13+
# uv run codeclash ladder make configs/ablations/ladder/make_scml.yaml --workers 30
14+
# Cheap SANITY pilot first (~1 h @ 32 cores): temporarily set sims_per_round: 30, run, rank, and
15+
# eyeball the ordering (baselines near the bottom) before committing to the full 400-sim pass.
16+
tournament:
17+
rounds: 0
18+
game:
19+
name: SCML
20+
sims_per_round: 400
21+
# sims_per_round: 30 # <- pilot: uncomment (and comment out 400) for a ~1h sanity-check ranking
22+
args:
23+
n_steps: 10
24+
n_lines: 2
25+
players:
26+
- agent: dummy
27+
branch_init: human/scml-baselines/greedy
28+
- agent: dummy
29+
branch_init: human/scml-baselines/nice
30+
- agent: dummy
31+
branch_init: human/scml-baselines/random
32+
- agent: dummy
33+
branch_init: human/scml2021/staghunter
34+
- agent: dummy
35+
branch_init: human/scml2021/team_50
36+
- agent: dummy
37+
branch_init: human/scml2021/team_51
38+
- agent: dummy
39+
branch_init: human/scml2021/team_54
40+
- agent: dummy
41+
branch_init: human/scml2021/team_55
42+
- agent: dummy
43+
branch_init: human/scml2021/team_61
44+
- agent: dummy
45+
branch_init: human/scml2021/team_62
46+
- agent: dummy
47+
branch_init: human/scml2021/team_72
48+
- agent: dummy
49+
branch_init: human/scml2021/team_73
50+
- agent: dummy
51+
branch_init: human/scml2021/team_86
52+
- agent: dummy
53+
branch_init: human/scml2021/team_90
54+
- agent: dummy
55+
branch_init: human/scml2021/team_corleone
56+
- agent: dummy
57+
branch_init: human/scml2022/team_102
58+
- agent: dummy
59+
branch_init: human/scml2022/team_103
60+
- agent: dummy
61+
branch_init: human/scml2022/team_105
62+
- agent: dummy
63+
branch_init: human/scml2022/team_106
64+
- agent: dummy
65+
branch_init: human/scml2022/team_107
66+
- agent: dummy
67+
branch_init: human/scml2022/team_123
68+
- agent: dummy
69+
branch_init: human/scml2022/team_124
70+
- agent: dummy
71+
branch_init: human/scml2022/team_126
72+
- agent: dummy
73+
branch_init: human/scml2022/team_131
74+
- agent: dummy
75+
branch_init: human/scml2022/team_134
76+
- agent: dummy
77+
branch_init: human/scml2022/team_94
78+
- agent: dummy
79+
branch_init: human/scml2022/team_96
80+
- agent: dummy
81+
branch_init: human/scml2023/team_102
82+
- agent: dummy
83+
branch_init: human/scml2023/team_123
84+
- agent: dummy
85+
branch_init: human/scml2023/team_126
86+
- agent: dummy
87+
branch_init: human/scml2023/team_127
88+
- agent: dummy
89+
branch_init: human/scml2023/team_134
90+
- agent: dummy
91+
branch_init: human/scml2023/team_143
92+
- agent: dummy
93+
branch_init: human/scml2023/team_144
94+
- agent: dummy
95+
branch_init: human/scml2023/team_145
96+
- agent: dummy
97+
branch_init: human/scml2023/team_148
98+
- agent: dummy
99+
branch_init: human/scml2023/team_149
100+
- agent: dummy
101+
branch_init: human/scml2023/team_151
102+
- agent: dummy
103+
branch_init: human/scml2023/team_poli_usp
104+
- agent: dummy
105+
branch_init: human/scml2024/coyoteteam
106+
- agent: dummy
107+
branch_init: human/scml2024/ozug4
108+
- agent: dummy
109+
branch_init: human/scml2024/team_144
110+
- agent: dummy
111+
branch_init: human/scml2024/team_164
112+
- agent: dummy
113+
branch_init: human/scml2024/team_171
114+
- agent: dummy
115+
branch_init: human/scml2024/team_172
116+
- agent: dummy
117+
branch_init: human/scml2024/team_174
118+
- agent: dummy
119+
branch_init: human/scml2024/team_193
120+
- agent: dummy
121+
branch_init: human/scml2024/team_238
122+
- agent: dummy
123+
branch_init: human/scml2024/team_abc
124+
- agent: dummy
125+
branch_init: human/scml2024/team_miyajima
126+
- agent: dummy
127+
branch_init: human/scml2024/teamyuzuru
128+
prompts:
129+
game_description: SCML OneShot ladder
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Arena smoke test (Phase 3) for the SCML ladder pilot: run the ported human bot
2+
# through the REAL arena (Docker build → clone CodeClash-ai/SCML → branch_init checkout
3+
# → validate_code → real SCML2024OneShot games) against a fallback-greedy dummy.
4+
# GITHUB_TOKEN=$(gh auth token) uv run codeclash run configs/ablations/ladder/scml_ffa.yaml
5+
tournament:
6+
rounds: 1
7+
game:
8+
name: SCML
9+
sims_per_round: 4
10+
args:
11+
n_steps: 8
12+
n_lines: 2
13+
players:
14+
- agent: dummy
15+
name: greedy-oneshot
16+
branch_init: human/scml-baselines/greedy-oneshot
17+
- agent: dummy
18+
name: baseline-fallback # default branch scml_agent.py -> returns {} -> greedy fallback
19+
prompts:
20+
game_description: SCML ladder smoke test
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Porting an SCML OneShot agent to the CodeClash `decide()` contract
2+
3+
You are porting ONE competition agent from `yasserfarouk/scml-agents` into a single
4+
self-contained `scml_agent.py` that defines **one function**: `decide(observation)`.
5+
A fully-worked, validated reference port lives at
6+
`configs/ablations/ladder/scml_pilot/scml_agent.py` (GreedyOneShotAgent) — read it first;
7+
mirror its structure and defensive style.
8+
9+
## The runtime (how your `decide` is called)
10+
11+
The trusted runtime owns a real SCML agent (base class `GreedySyncAgent`) and calls your
12+
`decide(observation)` at each negotiation turn. `observation` is a plain dict:
13+
14+
```
15+
observation = {
16+
"event": "propose" | "respond", # which decision is being asked
17+
"player": "<name>",
18+
"negotiator_id": "<id>" | None,
19+
"awi": { # agent-world-info (all plain numbers/lists)
20+
"current_step", "n_steps", "n_lines", "max_n_lines",
21+
"current_balance", "current_inventory",
22+
"current_exogenous_input_quantity", "current_exogenous_input_price",
23+
"current_exogenous_output_quantity", "current_exogenous_output_price",
24+
"current_disposal_cost", "current_shortfall_penalty",
25+
"my_input_product", "my_output_product",
26+
"is_first_level", "is_middle_level", "is_last_level",
27+
"needed_sales", "needed_supplies", # <-- USE THESE for "how much do I still need"
28+
"my_suppliers", "my_consumers",
29+
},
30+
"nmi": { # negotiation-mechanism-info
31+
"annotation": { "product": <int>, "buyer": <id>, "seller": <id>, ... },
32+
"issues": [ {"name","min","max","values"}, # [0]=QUANTITY, [1]=TIME, [2]=UNIT_PRICE
33+
{...}, {...} ],
34+
},
35+
"state": { "step": <int>, "relative_time": <float 0..1>, "current_offer": [q,t,up] | None },
36+
# on "respond": observation["current_offer"] and/or state["current_offer"] holds the opponent offer
37+
"fallback_offer": [q,t,up] | None, # what the greedy fallback would offer
38+
"fallback_response": "accept"|"reject"|"end",
39+
}
40+
```
41+
42+
### What to return
43+
- On `event == "propose"`: `{"offer": [quantity, time, unit_price]}` — three **ints**, each
44+
within its issue's `[min, max]`. Out-of-range/non-int → counted as a policy error, fallback used.
45+
- On `event == "respond"`: `{"response": "accept" | "reject" | "end"}`. When rejecting you MAY
46+
include a counter: `{"response": "reject", "offer": [q,t,up]}`.
47+
- Return `{}` or `None` at any point to defer to the trusted greedy fallback (safe default).
48+
49+
### Hard rules
50+
- **Never import `scml`, `negmas`, `numpy`, or anything outside the stdlib.** The port must be
51+
pure-Python stdlib only — you only have the observation dict. Re-express the strategy's math
52+
from scratch.
53+
- **Never raise.** Wrap the body in `try/except` returning `{}` (see reference). An unhandled
54+
exception floors the score.
55+
- `is_selling` = `nmi["annotation"].get("product") == awi.get("my_output_product")`.
56+
- "How much I still need" = `awi["needed_sales"]` if selling else `awi["needed_supplies"]`.
57+
- Concession over time: use `state["relative_time"]` (0 at start → 1 at deadline) instead of
58+
`state.step / nmi.n_steps` (n_steps isn't exposed on nmi).
59+
- No cross-call state is guaranteed; if the original kept per-step memory (opponent price models,
60+
`on_negotiation_success`), you may keep **module-level** dicts keyed by negotiator_id, but you
61+
get NO success/step callbacks — approximate or drop that refinement and note it in the docstring.
62+
63+
## Mapping the source's methods
64+
- source `propose(negotiator_id, state)` / `first_proposals` → your `event=="propose"` branch.
65+
- source `respond(...)` / `counter_all(offers, states)` → your `event=="respond"` branch.
66+
- source `best_offer` / price helpers (`_find_good_price`, `_price_range`, `_th`) → inline as
67+
plain functions over the observation (reference port shows the pattern).
68+
69+
## If the bot is RL / learned / infeasible
70+
Some agents (q-learning, PPO, regression on trained weights) can't be reproduced without their
71+
weights/deps. In that case: port the **heuristic core** if one exists (many have a rule-based
72+
path); otherwise DO NOT fake it — return a short note in your report that the bot is
73+
RL-weight-dependent and should be skipped, and still write a best-effort heuristic port only if
74+
faithful. Log the reason.
75+
76+
## Deliverable
77+
Write your port to `configs/ablations/ladder/scml_pilot/ports/<YEAR>__<TEAM>.py`
78+
(e.g. `scml2024__team_193.py`). It must `python3 -c "import ast; ast.parse(open(FILE).read())"`
79+
cleanly and define a top-level `decide`. Keep a concise module docstring naming the source and
80+
noting any simplifications/dropped features.
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Running the SCML round-robin (Elo ranking) on AWS
2+
3+
Goal: run the 1,275-pair round-robin in `make_scml.yaml` to rank the 51 human bots, then
4+
compute Elo and assemble the ladder. The 51 bots already live on `CodeClash-ai/SCML` as
5+
`human/*` branches — `branch_init` fetches them at runtime, so nothing bot-side needs shipping;
6+
you only need this repo branch + Docker + a GitHub token.
7+
8+
## Prerequisites on the AWS box
9+
- Docker running (`docker info`), git, and `uv` (repo uses `uv run codeclash ...`).
10+
- A GitHub token with read access to `CodeClash-ai/SCML` (public, so a default `gh auth token`
11+
or any classic PAT works). Export it as `GITHUB_TOKEN` for the run.
12+
- This branch pulled: `git fetch && git checkout <branch> && uv sync` (or the repo's usual setup).
13+
14+
## Step 0 — pre-build the arena image ONCE (avoids a build stampede)
15+
`ladder make --workers N` builds the image lazily per pair; with many workers they'd all try to
16+
build at once. Build it a single time up front (it `git clone`s CodeClash-ai/SCML and installs
17+
`scml==0.8.2`):
18+
19+
```bash
20+
docker build -t codeclash/scml -f codeclash/arenas/scml/SCML.Dockerfile .
21+
```
22+
23+
Sanity check one pair end-to-end before the big run:
24+
25+
```bash
26+
bash configs/ablations/ladder/scml_pilot/smoke_scml.sh # greedy vs dummy, should print PASS
27+
```
28+
29+
## Step 1 (recommended) — cheap pilot ranking (~1 h on 32 cores)
30+
Edit `configs/ablations/ladder/make_scml.yaml`: comment out `sims_per_round: 400`, uncomment
31+
`sims_per_round: 30`. Then:
32+
33+
```bash
34+
GITHUB_TOKEN=$(gh auth token) \
35+
uv run codeclash ladder make configs/ablations/ladder/make_scml.yaml --workers 30
36+
python -m codeclash.analysis.metrics.elo -d logs/ladder/SCML --output-dir assets/scml_elo_pilot
37+
```
38+
39+
Eyeball the printed Elo ordering: the baselines (`nice` < `random` < `greedy`) should sit near
40+
the bottom, and disciplined bots should outrank very concessive ones. If it looks sane, proceed.
41+
(The pilot logs live in a different pair-count than the full run only in `sims`; to force fresh
42+
full-sim logs, run the full pass in a clean `logs/` or a separate `-o` dir — see note below.)
43+
44+
## Step 2 — full ranking run (400 sims, ~23 h on 32 cores)
45+
Restore `sims_per_round: 400` in the config, then launch under `nohup`/`tmux` so it survives
46+
disconnects:
47+
48+
```bash
49+
tmux new -s scml
50+
GITHUB_TOKEN=$(gh auth token) \
51+
uv run codeclash ladder make configs/ablations/ladder/make_scml.yaml --workers 30 \
52+
2>&1 | tee scml_make.log
53+
# detach: Ctrl-b d | reattach: tmux attach -t scml
54+
```
55+
56+
- **Resumable:** each pair writes to `logs/ladder/SCML/PvpTournament.<a>_vs_<b>/`; a rerun skips
57+
pairs whose folder already exists. Safe to stop/restart. If it dies, just relaunch the same
58+
command — it continues where it left off.
59+
- **`--workers 30`** on a 32-core box (leave 2 cores headroom; the `decide` 3s timeout can trip
60+
under CPU oversubscription). Progress = count of `logs/ladder/SCML/PvpTournament.*` dirs
61+
(target 1275): `ls -d logs/ladder/SCML/PvpTournament.* | wc -l`.
62+
63+
> NOTE on pilot→full log mixing: if you ran the Step-1 pilot into `logs/`, move or delete
64+
> `logs/ladder/SCML/` before the full run (or point the pilot elsewhere) so the Elo fit uses only
65+
> the 400-sim results. The make command has no `-o`; it always writes under `logs/ladder/<Arena>/`.
66+
67+
## Step 3 — compute Elo and rank
68+
```bash
69+
python -m codeclash.analysis.metrics.elo -d logs/ladder/SCML --output-dir assets/scml_elo
70+
```
71+
Prints the Bradley-Terry/Elo ranking and writes `assets/scml_elo/elo_results.log` (+ plots,
72+
LaTeX/website tables). The ordering is what becomes the ladder (weakest → strongest).
73+
74+
## Step 4 — assemble the ladder configs (do this once you have the ranking)
75+
1. `configs/ablations/ladder/rungs/scml.yaml` — the ranked opponent list, worst first, strongest
76+
last (each `{agent: dummy, branch_init: human/...}`), in Elo order from Step 3.
77+
2. `configs/ablations/ladder/scml.yaml` — the run config: a climbing `player` (starting at the
78+
weakest rung) + `ladder: !include ablations/ladder/rungs/scml.yaml` + a `ladder_rules` block.
79+
Model on `battlesnake.yaml`.
80+
3. Optional `scml__<model>.yaml` per-model variants (swap `model: !include mini/models/...`).
81+
82+
(Ping me with `logs/ladder/SCML/` or the `elo_results.log` and I'll generate the rungs + run
83+
configs automatically.)
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
"""Dummy opponent: always defer to the trusted greedy fallback.
2+
3+
Returning {} at every turn means the runtime's built-in GreedySyncAgent drives this
4+
player. Used as the baseline opponent in the pilot smoke test.
5+
"""
6+
7+
8+
def decide(observation):
9+
return {}

0 commit comments

Comments
 (0)