Skip to content

Commit 34fbc84

Browse files
timodonnellclaude
andcommitted
exp4 rerun @ 5 seeds × 5 samples + default n_seeds=5
exp8 round 3 showed that single-seed diffusion gives abnormally narrow sample distributions — on 8t59 the DockQ ≥ 0.23 success rate in the sample pool went from 44% (1 seed × 25) to 92% (5 seeds × 5). This likely affects categories beyond ab-ag too. Changes: - ensure_bench_run default n_seeds=5 (was 1). Old behavior accessible via explicit n_seeds=1. est_wall_hours default bumped to 2h to reflect 5× recycling overhead. - exp4 notebook: new run name `protenix-v1-5seeds`, n_samples=5 + n_seeds=5, est_wall_hours=4. Previous single-seed run remains cached under `protenix-v1-default` for reference. Estimated cost ~$126 (over $100 gate; run approved by user as the new baseline against which all future experiments compare). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 290602c commit 34fbc84

2 files changed

Lines changed: 11 additions & 5 deletions

File tree

experiments/exp4_baseline_protenix_v1/README.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,20 +66,26 @@ Expected wall clock ~45 min; estimated cost ~$24.
6666

6767
```python
6868
bench = ensure_bench_run(
69-
"protenix-v1-default",
69+
"protenix-v1-5seeds",
7070
checkpoint="protenix-v1",
7171
workers=8,
7272
gpu="H100",
73-
n_samples=5,
73+
n_samples=5, # samples per seed
74+
n_seeds=5, # total predictions = 25, matches published FoldBench protocol
7475
n_cycles=10,
7576
max_tokens=2048,
7677
cutoff_date="2024-01-01",
77-
est_wall_hours=0.75,
78+
est_wall_hours=4.0, # 5× recycling per target ⇒ ~5× 0.75h
7879
)
7980
git_sha = bench.meta.get("git_sha") or "?"
8081
print(f"cached: {bench.cached} | git_sha: {git_sha[:8]}")
8182
```
8283

84+
This is the new baseline. The previous single-seed run
85+
(`protenix-v1-default`) is kept under the old `data/summary.csv` for
86+
reference; the n=5×5 numbers become the post-exp8-round-3 canonical
87+
comparison point.
88+
8389
## Summary table
8490

8591
```python

src/helico/experiment.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -304,13 +304,13 @@ def ensure_bench_run(
304304
workers: int = 8,
305305
gpu: str = "H100",
306306
n_samples: int = 5,
307-
n_seeds: int = 1,
307+
n_seeds: int = 5,
308308
max_tokens: int = 2048,
309309
n_cycles: int = 10,
310310
cutoff_date: str = "2024-01-01",
311311
categories: str = "",
312312
target_pdb_ids: str = "",
313-
est_wall_hours: float = 0.5,
313+
est_wall_hours: float = 2.0,
314314
force: bool = False,
315315
publish: bool = False,
316316
) -> BenchRun:

0 commit comments

Comments
 (0)