Skip to content

Commit a006ba5

Browse files
constantinpapeconstantinpape
committed
Add benchmark results
1 parent 569ed22 commit a006ba5

2 files changed

Lines changed: 149 additions & 22 deletions

File tree

‎development/graph/multicut/PERFORMANCE_NOTES.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Multicut Performance Notes
2+
3+
State of the multicut solvers vs nifty on the standard benchmark problems and
4+
notes on remaining optimization headroom. Read this before the next round of
5+
perf work.
6+
7+
## Current benchmark matrix
8+
9+
Produced by `python evaluate_solvers.py` (2026-05-17). Small problems were run
10+
with both implementations in one pass:
11+
12+
```bash
13+
python evaluate_solvers.py --problems A_small B_small C_small \
14+
--results-jsonl benchmark_results/small_both.jsonl
15+
```
16+
17+
Medium problems were run in two passes so completed rows were preserved even if
18+
a long nifty row had to be stopped:
19+
20+
```bash
21+
python evaluate_solvers.py --problems A_medium B_medium C_medium \
22+
--backend bic --results-jsonl benchmark_results/medium_bic.jsonl
23+
python evaluate_solvers.py --problems A_medium B_medium C_medium \
24+
--backend nifty --results-jsonl benchmark_results/medium_nifty.jsonl
25+
```
26+
27+
All runs are single-threaded with `n_repeats=1`. `KernighanLinMulticut` and the
28+
KL stage inside `ChainedMulticutSolvers` use 5 outer iterations. Fusion-move
29+
uses watershed proposals, `numberOfIterations=10`, `stopIfNoImprovement=4`, and
30+
a greedy-additive fusion sub-solver on the nifty side. `runtime ratio` is
31+
`nifty_runtime / bic_runtime` - values reported as "faster" mean bic is faster.
32+
33+
| Problem | Solver | bic energy | nifty energy | Δenergy | bic runtime | nifty runtime | runtime ratio |
34+
|---|---|---|---|---|---|---|---|
35+
| A_small | greedy_additive | -76 914.52 | -76 914.52 | 0.00 | 0.28 s | 0.38 s | 1.32x faster |
36+
| A_small | kernighan_lin | -76 916.06 | -76 916.06 | 0.00 | 2.04 s | 2.54 s | 1.24x faster |
37+
| A_small | greedy_fixation | -76 914.13 | -76 914.13 | 0.00 | 0.28 s | 1.96 s | 7.09x faster |
38+
| A_small | chained | -76 916.06 | -76 916.06 | 0.00 | 2.02 s | 2.56 s | 1.27x faster |
39+
| A_small | decomposer | -76 914.52 | -76 914.52 | 0.00 | 0.26 s | 0.34 s | 1.30x faster |
40+
| A_small | fusion_move | -76 915.29 | -76 915.29 | 0.00 | 1.60 s | 2.35 s | 1.46x faster |
41+
| B_small | greedy_additive | -437 001.4 | -437 001.4 | 0.00 | 0.32 s | 0.44 s | 1.36x faster |
42+
| B_small | kernighan_lin | -437 023.6 | -437 023.6 | 0.00 | 4.40 s | 5.32 s | 1.21x faster |
43+
| B_small | greedy_fixation | -436 943.8 | -436 943.8 | 0.00 | 0.34 s | 1.86 s | 5.43x faster |
44+
| B_small | chained | -437 023.6 | -437 023.6 | 0.00 | 4.36 s | 5.27 s | 1.21x faster |
45+
| B_small | decomposer | -437 001.4 | -437 001.4 | 0.00 | 0.32 s | 0.49 s | 1.50x faster |
46+
| B_small | fusion_move | -437 034.1 | -437 034.1 | 0.00 | 1.60 s | 2.62 s | 1.64x faster |
47+
| C_small | greedy_additive | -24 189.93 | -24 189.93 | 0.00 | 0.74 s | 0.87 s | 1.17x faster |
48+
| C_small | kernighan_lin | -24 191.91 | -24 191.95 | +0.03698 | 6.52 s | 7.41 s | 1.14x faster |
49+
| C_small | greedy_fixation | -24 162.43 | -24 162.43 | 0.00 | 0.76 s | 2.79 s | 3.66x faster |
50+
| C_small | chained | -24 191.91 | -24 191.95 | +0.03698 | 6.52 s | 7.33 s | 1.12x faster |
51+
| C_small | decomposer | -24 189.93 | -24 189.93 | 0.00 | 0.74 s | 0.91 s | 1.23x faster |
52+
| C_small | fusion_move | -24 191.45 | -24 191.45 | 0.00 | 4.90 s | 7.42 s | 1.52x faster |
53+
| A_medium | greedy_additive | -535 228.2 | -535 228.2 | 0.00 | 6.21 s | 6.72 s | 1.08x faster |
54+
| A_medium | kernighan_lin | -535 251.2 | -535 251.3 | +0.0821 | 124.75 s | 137.72 s | 1.10x faster |
55+
| A_medium | greedy_fixation | -535 214.6 | -535 214.6 | 0.00 | 6.16 s | 25.26 s | 4.10x faster |
56+
| A_medium | chained | -535 251.2 | -535 251.3 | +0.0821 | 122.41 s | 134.68 s | 1.10x faster |
57+
| A_medium | decomposer | -535 228.2 | -535 228.2 | 0.00 | 6.15 s | 5.81 s | 1.06x slower |
58+
| A_medium | fusion_move | -535 235.3 | -535 235.3 | 0.00 | 51.34 s | 65.61 s | 1.28x faster |
59+
| B_medium | greedy_additive | -2 141 349 | -2 141 349 | 0.00 | 5.71 s | 6.78 s | 1.19x faster |
60+
| B_medium | kernighan_lin | -2 141 705 | -2 141 700 | -5.246 | 263.46 s | 157.91 s | 1.67x slower |
61+
| B_medium | greedy_fixation | -2 141 097 | -2 141 097 | 0.00 | 5.63 s | 24.22 s | 4.30x faster |
62+
| B_medium | chained | -2 141 705 | -2 141 700 | -5.246 | 259.35 s | 156.60 s | 1.66x slower |
63+
| B_medium | decomposer | -2 141 349 | -2 141 349 | 0.00 | 5.62 s | 7.00 s | 1.25x faster |
64+
| B_medium | fusion_move | -2 141 572 | -2 141 572 | 0.00 | 43.82 s | 67.12 s | 1.53x faster |
65+
| C_medium | greedy_additive | -90 732.51 | -90 732.51 | 0.00 | 11.05 s | 11.53 s | 1.04x faster |
66+
| C_medium | kernighan_lin | -90 768.01 | -90 768.56 | +0.5491 | 219.36 s | 240.26 s | 1.10x faster |
67+
| C_medium | greedy_fixation | -90 631.67 | -90 631.67 | 0.00 | 11.11 s | 30.28 s | 2.73x faster |
68+
| C_medium | chained | -90 768.01 | -90 768.56 | +0.5491 | 218.77 s | 234.48 s | 1.07x faster |
69+
| C_medium | decomposer | -90 732.51 | -90 732.51 | 0.00 | 10.84 s | 12.12 s | 1.12x faster |
70+
| C_medium | fusion_move | -90 746.68 | -90 746.68 | 0.00 | 58.34 s | 86.42 s | 1.48x faster |
71+
72+
Δenergy = bic - nifty; negative means bic found the lower-energy labeling,
73+
positive means nifty did. The only nonzero differences are KL/chained rows, and
74+
they are tiny relative to the objective scale. Fusion-move, greedy-additive,
75+
greedy-fixation, and decomposer match nifty energies on every benchmark row.
76+
77+
## Current read
78+
79+
`KernighanLinMulticut` is the dominant runtime target. It is faster than nifty
80+
on most rows, but `B_medium` is the clear exception: bic KL/chained takes about
81+
260 s vs nifty's 157 s. Because `ChainedMulticutSolvers` is greedy-additive +
82+
KL, it inherits the same behavior.
83+
84+
Fusion-move is not the immediate bottleneck. It is faster than nifty on every
85+
row in the matrix, including all medium problems, with exact energy matches.
86+
87+
Greedy fixation is substantially faster than nifty, and greedy additive plus
88+
the decomposer are already close enough that further work there is unlikely to
89+
move end-to-end runtimes unless a downstream workflow uses only those solvers.
90+
91+
Raw intermediate rows are in `benchmark_results/*.jsonl`; the `.md` files next
92+
to them contain the direct stdout tables from each run.

‎development/graph/multicut/evaluate_solvers.py‎

Lines changed: 57 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from __future__ import annotations
22

33
import argparse
4+
import json
45
import sys
56
from dataclasses import dataclass
67
from statistics import mean
@@ -142,39 +143,54 @@ def evaluate(problem_name: str, solver_name: str, config: SolverConfig, args):
142143
nifty_runtimes = []
143144

144145
for _ in range(args.n_repeats):
145-
bic_objective = bic.graph.MulticutObjective(bic_graph, costs)
146-
start = perf_counter()
147-
bic_labels = config.make_bic_solver(args.threads).optimize(bic_objective)
148-
bic_runtimes.append(perf_counter() - start)
149-
bic_energies.append(bic_energy(bic_graph, costs, bic_labels))
150-
151-
nifty_objective = nmc.multicutObjective(nifty_graph, costs)
152-
start = perf_counter()
153-
nifty_labels = (
154-
config.make_nifty_factory(nifty_objective, args.threads)
155-
.create(nifty_objective)
156-
.optimize()
157-
)
158-
nifty_runtimes.append(perf_counter() - start)
159-
nifty_energies.append(nifty_energy(nifty_graph, costs, nifty_labels))
160-
161-
bic_runtime = mean(bic_runtimes)
162-
nifty_runtime = mean(nifty_runtimes)
146+
if args.backend in ("both", "bic"):
147+
bic_objective = bic.graph.MulticutObjective(bic_graph, costs)
148+
start = perf_counter()
149+
bic_labels = config.make_bic_solver(args.threads).optimize(bic_objective)
150+
bic_runtimes.append(perf_counter() - start)
151+
bic_energies.append(bic_energy(bic_graph, costs, bic_labels))
152+
153+
if args.backend in ("both", "nifty"):
154+
nifty_objective = nmc.multicutObjective(nifty_graph, costs)
155+
start = perf_counter()
156+
nifty_labels = (
157+
config.make_nifty_factory(nifty_objective, args.threads)
158+
.create(nifty_objective)
159+
.optimize()
160+
)
161+
nifty_runtimes.append(perf_counter() - start)
162+
nifty_energies.append(nifty_energy(nifty_graph, costs, nifty_labels))
163+
164+
bic_runtime = mean(bic_runtimes) if bic_runtimes else None
165+
nifty_runtime = mean(nifty_runtimes) if nifty_runtimes else None
166+
bic_energy_value = mean(bic_energies) if bic_energies else None
167+
nifty_energy_value = mean(nifty_energies) if nifty_energies else None
168+
energy_diff = (
169+
bic_energy_value - nifty_energy_value
170+
if bic_energy_value is not None and nifty_energy_value is not None
171+
else None
172+
)
163173
return {
164174
"problem": problem_name,
165175
"solver": solver_name,
166176
"nodes": int(bic_graph.number_of_nodes),
167177
"edges": int(bic_graph.number_of_edges),
168-
"bic_energy": mean(bic_energies),
169-
"nifty_energy": mean(nifty_energies),
170-
"energy_diff": mean(bic_energies) - mean(nifty_energies),
178+
"bic_energy": bic_energy_value,
179+
"nifty_energy": nifty_energy_value,
180+
"energy_diff": energy_diff,
171181
"bic_runtime_s": bic_runtime,
172182
"nifty_runtime_s": nifty_runtime,
173-
"runtime_ratio": nifty_runtime / bic_runtime if bic_runtime > 0 else float("inf"),
183+
"runtime_ratio": (
184+
nifty_runtime / bic_runtime
185+
if bic_runtime is not None and nifty_runtime is not None and bic_runtime > 0
186+
else None
187+
),
174188
}
175189

176190

177191
def format_float(value: float) -> str:
192+
if value is None:
193+
return ""
178194
return f"{value:.6g}"
179195

180196

@@ -194,6 +210,13 @@ def print_progress(row: dict) -> None:
194210
)
195211

196212

213+
def append_jsonl(path: str, row: dict) -> None:
214+
with open(path, "a", encoding="utf-8") as f:
215+
json.dump(row, f, sort_keys=True)
216+
f.write("\n")
217+
f.flush()
218+
219+
197220
def print_markdown_table(rows: list[dict]) -> None:
198221
headers = [
199222
"problem",
@@ -250,6 +273,16 @@ def main() -> None:
250273
parser.add_argument("--n-repeats", type=int, default=1)
251274
parser.add_argument("--threads", type=int, default=1)
252275
parser.add_argument("--timeout", type=float, default=30.0)
276+
parser.add_argument(
277+
"--backend",
278+
choices=("both", "bic", "nifty"),
279+
default="both",
280+
help="Which implementation to run. Defaults to both.",
281+
)
282+
parser.add_argument(
283+
"--results-jsonl",
284+
help="Append each completed row to this JSONL file.",
285+
)
253286
args = parser.parse_args()
254287

255288
if args.n_repeats < 1:
@@ -263,6 +296,8 @@ def main() -> None:
263296
row = evaluate(problem_name, solver_name, configs[solver_name], args)
264297
rows.append(row)
265298
print_progress(row)
299+
if args.results_jsonl is not None:
300+
append_jsonl(args.results_jsonl, row)
266301
print_markdown_table(rows)
267302

268303

0 commit comments

Comments
 (0)