-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
1026 lines (982 loc) · 54.9 KB
/
Copy pathindex.html
File metadata and controls
1026 lines (982 loc) · 54.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>WeaveBench | Long-Horizon Hybrid-Interface Benchmark for Computer-Use Agents</title>
<meta name="description" content="WeaveBench: 114 real-world tasks across 8 work domains where computer-use agents must interleave GUI control with CLI/code execution within a single trajectory. Evaluated by a trajectory-aware judge that detects shortcut behaviors. Best frontier pairing reaches only 41.2% PassRate.">
<meta name="keywords" content="Computer-Use Agents, CUA, GUI agent, CLI agent, hybrid interface, long-horizon, benchmark, OSWorld, OpenClaw, Codex CLI, Claude Code, Hermes, Agent-as-a-Judge, trajectory grading, reward hacking">
<meta property="og:title" content="WeaveBench: Long-Horizon Hybrid GUI+CLI Benchmark for Computer-Use Agents">
<meta property="og:description" content="114 tasks, 8 domains, 4 deployed agent harnesses. Even the best frontier pairing reaches only 41.2% PassRate.">
<meta property="og:image" content="static/images/casestudy_3panel.png">
<meta property="og:type" content="website">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="WeaveBench">
<meta name="twitter:description" content="A long-horizon hybrid GUI+CLI benchmark for computer-use agents. Best frontier pairing: 41.2% PassRate.">
<meta name="twitter:image" content="static/images/casestudy_3panel.png">
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 23 23'%3E%3Crect width='10' height='10' fill='%23F25022'/%3E%3Crect x='13' width='10' height='10' fill='%237FBA00'/%3E%3Crect y='13' width='10' height='10' fill='%2300A4EF'/%3E%3Crect x='13' y='13' width='10' height='10' fill='%23FFB900'/%3E%3C/svg%3E">
<link rel="stylesheet" href="static/css/index.css?v=20260722a">
</head>
<body>
<header class="topbar" id="navbar" aria-label="Page navigation">
<div class="navbar-logos">
<a class="brandmark" href="#top" aria-label="WeaveBench home">
<svg class="microsoft-mark" viewBox="0 0 23 23" aria-hidden="true">
<rect width="10" height="10" fill="#F25022"/>
<rect x="13" width="10" height="10" fill="#7FBA00"/>
<rect y="13" width="10" height="10" fill="#00A4EF"/>
<rect x="13" y="13" width="10" height="10" fill="#FFB900"/>
</svg>
<span class="brand-weave">Weave</span><span class="brand-bench">Bench</span>
</a>
<div class="navbar-divider" aria-hidden="true"></div>
<a class="navbar-related" href="https://github.com/weavebench/WeaveBench" target="_blank" rel="noopener" title="WeaveBench code repository">
<span class="nr-icon" aria-hidden="true"></span>
<span>
<span class="nr-label">Release</span>
<span class="nr-name">v0.1 · 114 tasks</span>
</span>
</a>
</div>
<nav class="nav" aria-label="Sections">
<a href="#idea">Idea</a>
<a href="#pipeline">Pipeline</a>
<a href="#dataset">Dataset</a>
<a href="#demos">Demos</a>
<a class="nav-leaderboard" href="#leaderboard">🏆 Leaderboard</a>
<a href="#ablations">Ablations</a>
<a href="#analysis">Analysis</a>
<a href="#citation">Cite</a>
<a href="trajectories/">Trajectories</a>
<a href="https://github.com/weavebench/WeaveBench" target="_blank" rel="noopener">Code</a>
</nav>
</header>
<section class="hero" id="top">
<div class="hero-inner">
<div>
<span class="kicker">A long-horizon, real-world benchmark for computer-use agents with hybrid interfaces</span>
<h1>WeaveBench</h1>
<p class="hero-subtitle">
<strong>114 tasks</strong> across <strong>8 real-world work domains</strong>, each requiring an agent to
<em>weave</em> GUI observation with CLI/code execution within a single trajectory.
Rollouts run inside a real Ubuntu sandbox; a trajectory-aware Agent-as-a-Judge
audits eight quality dimensions and zeros credit on any of nine shortcut
patterns (synthetic screenshots, hard-coded metrics, mock services…).
</p>
<div class="hero-actions" aria-label="Primary links">
<a class="button primary" href="#idea">Core Idea</a>
<a class="button secondary" href="#pipeline">Pipeline</a>
<a class="button secondary" href="#results">Results</a>
<a class="button tertiary" href="https://github.com/weavebench/WeaveBench" target="_blank" rel="noopener">
<svg class="button-icon" viewBox="0 0 16 16" aria-hidden="true">
<path d="M8 0C3.58 0 0 3.67 0 8.2c0 3.62 2.29 6.69 5.47 7.78.4.08.55-.18.55-.4 0-.2-.01-.86-.01-1.56-2.01.38-2.53-.5-2.69-.95-.09-.23-.48-.95-.82-1.14-.28-.15-.68-.52-.01-.53.63-.01 1.08.59 1.23.83.72 1.24 1.87.89 2.33.68.07-.53.28-.89.51-1.09-1.78-.21-3.64-.91-3.64-4.03 0-.89.31-1.62.82-2.19-.08-.21-.36-1.04.08-2.16 0 0 .67-.22 2.2.84A7.42 7.42 0 0 1 8 4.01c.68 0 1.36.09 2 .27 1.53-1.06 2.2-.84 2.2-.84.44 1.12.16 1.95.08 2.16.51.57.82 1.3.82 2.19 0 3.13-1.87 3.82-3.65 4.03.29.26.54.75.54 1.51 0 1.09-.01 1.97-.01 2.24 0 .22.15.48.55.4A8.1 8.1 0 0 0 16 8.2C16 3.67 12.42 0 8 0Z"/>
</svg>
Code Repo
</a>
<a class="button secondary" href="https://arxiv.org/abs/2606.09426" target="_blank" rel="noopener">
<svg class="button-icon" viewBox="0 0 16 16" aria-hidden="true">
<path d="M4 0a2 2 0 0 0-2 2v12a2 2 0 0 0 2 2h8a2 2 0 0 0 2-2V5l-5-5H4Zm5 1.5V5h3.5L9 1.5ZM5 9h6v1H5V9Zm0 2h6v1H5v-1Z"/>
</svg>
arXiv:2606.09426
</a>
<a class="button secondary" href="https://huggingface.co/datasets/wanlilll/WeaveBench" target="_blank" rel="noopener">
🤗 Dataset
</a>
<a class="button secondary" href="trajectories/">
🧵 Trajectories
</a>
</div>
<a class="related-project" href="https://github.com/weavebench/WeaveBench" target="_blank" rel="noopener" aria-label="Open the WeaveBench code repository">
<span class="related-icon" aria-hidden="true"></span>
<span class="related-text">
<span class="related-tag">One-command setup</span>
<span class="related-title"><strong>bash scripts/setup.sh</strong></span>
<span class="related-summary">Installs deps + downloads dataset + 28 GB qcow2.</span>
</span>
<span class="related-arrow" aria-hidden="true">→</span>
</a>
</div>
<aside class="hero-ledger" aria-label="Headline result">
<span class="ledger-kicker">Best frontier PassRate</span>
<div class="ledger-hero" aria-label="41.2 percent best frontier pairing">
<span class="ledger-value">41.2</span>
<span class="ledger-denominator">%</span>
</div>
<p class="ledger-copy">
Claude Opus 4.7 inside <strong>Claude Code</strong> — the strongest of any model×harness pairing we evaluated. Frontier backbones report >78% on the same generation of <em>OSWorld-Verified</em>.
</p>
<div class="ledger-stats" aria-label="Benchmark coverage">
<div class="ledger-stat"><span>Tasks</span><b>114</b></div>
<div class="ledger-stat"><span>Domains</span><b>8</b></div>
<div class="ledger-stat ledger-stat-harness"><span>Harnesses</span>
<div class="harness-logos" aria-label="OpenClaw, Codex CLI, Claude Code, Hermes">
<span class="harness-logo" title="OpenClaw"><img src="static/images/harness/openclaw.png?v=20260605a" alt="OpenClaw" loading="lazy"><b>OpenClaw</b></span>
<span class="harness-logo" title="Codex CLI"><img src="static/images/harness/codex.png?v=20260605b" alt="Codex CLI" loading="lazy"><b>Codex CLI</b></span>
<span class="harness-logo" title="Claude Code"><img src="static/images/harness/claudecode.png?v=20260605b" alt="Claude Code" loading="lazy"><b>Claude Code</b></span>
<span class="harness-logo" title="Hermes"><img src="static/images/harness/hermes.png?v=20260605a" alt="Hermes" loading="lazy"><b>Hermes</b></span>
</div>
</div>
</div>
</aside>
</div>
</section>
<main>
<!-- ============ Paper Teaser (Figure 1) ============ -->
<section class="teaser-showcase" id="teaser" aria-labelledby="teaser-title">
<div class="teaser-heading">
<span>Figure 1 from the paper</span>
<div>
<h2 id="teaser-title">Three real-world workflows that require interleaved GUI & CLI.</h2>
<p class="section-lede">
<strong>(DAV)</strong> Diagnose a Jaeger trace span by inspecting its
shape, then patch the upstream timeout via <code>kubectl</code>.
<strong>(GAM)</strong> Play a desktop game to localize a sprite/physics
bug, then patch the scene-graph source.
<strong>(OPS)</strong> Catch a 503 spike on a Web Ops dashboard, edit
<code>nginx.conf</code>, and re-check the dashboard.
Each step alternates between a GUI signal that no API exposes and a
CLI/code change that no screenshot can produce.
</p>
</div>
</div>
<figure class="teaser-figure">
<img src="static/images/casestudy_3panel.png?v=20260604d" alt="Three example WeaveBench workflows showing GUI and CLI steps side by side for DAV, GAM, and OPS domains.">
</figure>
</section>
<!-- ============ 01 CORE IDEA ============ -->
<section class="section" id="idea">
<div class="section-header">
<div class="section-eyebrow">01 / Core Idea</div>
<div>
<h2>Evaluate cross-interface orchestration, not isolated capabilities.</h2>
<p class="section-lede">
Deployed CUA runtimes already integrate GUI control, CLI/code
execution, browsers, and external tools inside one agent loop. Existing
benchmarks test these channels in isolation, leaving the orchestration
layer under-measured. WeaveBench is built around tasks where success
requires the agent to weave both channels together.
</p>
</div>
</div>
<div class="manifesto">
<article class="statement">
<h3>GUI and CLI are complementary, not interchangeable.</h3>
<p>
GUIs expose rendered, transient, spatial state — canvases,
dialogs, visual feedback. CLI/code expose structured, scriptable,
persistent state — source files, configs, logs, services. Real
workflows braid them: observe in the GUI, verify in the shell, edit in
code, re-render, re-observe.
</p>
<div class="chip-row">
<span class="chip">Real Ubuntu desktop</span>
<span class="chip">114 sandboxed tasks</span>
<span class="chip">8 work domains</span>
<span class="chip">4 deployed harnesses</span>
<span class="chip">Trajectory-aware judge</span>
</div>
</article>
<div class="steps" aria-label="Benchmark design pillars">
<div class="step">
<strong>P1 · Channel non-substitutability</strong>
<p>Each admitted task requires coordinating GUI observation/action with CLI/code modification inside the same trajectory; annotated with single-channel-bound atomic operations.</p>
</div>
<div class="step">
<strong>P2 · Long-horizon execution</strong>
<p>Expert reference trajectories contain multiple interleaved GUI and CLI/code phases — not a single perception, action, or tool-use step.</p>
</div>
<div class="step">
<strong>P3 · Cross-application state</strong>
<p>Tasks span multiple independent applications/processes whose states are linked by the workflow; agents must preserve and transfer information across them.</p>
</div>
<div class="step">
<strong>Trajectory-aware grading</strong>
<p>Final-only grading is fragile here. The judge audits transcripts, files, screenshots, and logs; nine shortcut detectors zero credit on confirmed reward hacks.</p>
</div>
</div>
</div>
</section>
<!-- ============ 02 PIPELINE (Method) ============ -->
<section class="section" id="pipeline">
<div class="section-header">
<div class="section-eyebrow">02 / Pipeline</div>
<div>
<h2>Task · Harness · Evaluation — one diagram, three pillars.</h2>
<p class="section-lede">
<strong>Task:</strong> 114 tasks across 8 domains, harvested from real
venues, packaged as
<code>ℰ = (𝒫, ℳ, 𝒞)</code>
bundles, audited against P1–P3, and stress-tested by ≥3 pilot
agents.
<strong>Harness:</strong> the agent runs in a single session over an
Ubuntu sandbox, with a minimal GUI plugin (one <code>screenshot</code>
tool + nine actuation primitives) layered on top of OpenClaw's CLI/code
tools; the same plugin is ported to Codex CLI, Claude Code,
and Hermes.
<strong>Evaluation</strong> is performed by an isolated
trajectory-aware agentic judge that combines bottom-up rubric scoring
with shortcut detection.
</p>
</div>
</div>
<figure class="figure-frame">
<img src="static/images/overview_new.png?v=20260604d" alt="WeaveBench pipeline: Task construction, hybrid harness with GUI plugin on top of CLI/code tools, and a trajectory-aware agentic judge.">
<figcaption class="caption">
<strong>Figure 2 from the paper.</strong>
Task construction (C1 archetype-guided sourcing → C4 pilot validation), the hybrid harness shared by all four runtimes, and the isolated trajectory-aware judge that re-fetches evidence across artifacts, screenshots, and logs.
</figcaption>
</figure>
<div class="method-grid">
<article class="method-card accent-blue">
<span class="method-num">M1</span>
<h3>Minimal GUI plugin (10 tools)</h3>
<p>
One perception primitive <code>screenshot</code> plus nine
<code>pyautogui</code>-backed actuation primitives
(<code>click</code>, <code>double_click</code>,
<code>triple_click</code>, <code>move</code>, <code>drag</code>,
<code>scroll</code>, <code>type</code>, <code>keypress</code>,
<code>wait</code>). Exposed alongside each runtime's terminal, file,
code, and browser tools — model loop and prompts unchanged.
</p>
</article>
<article class="method-card accent-red">
<span class="method-num">M2</span>
<h3>Trajectory-aware Agent-as-a-Judge</h3>
<p>
For every rollout, an isolated subprocess judge re-fetches evidence
over multiple turns using file, image, and shell tools; decomposes each
deliverable into atomic clauses; verifies each clause with cited
evidence; and assigns scores along eight process & outcome
dimensions.
</p>
</article>
<article class="method-card accent-gold">
<span class="method-num">M3</span>
<h3>Nine shortcut detectors</h3>
<p>
A parallel scan covers fake screenshots/renders, regenerated fixtures,
hard-coded metrics, mock services, duplicate crops, overlay
manipulation, ground-truth leakage, runtime injection, and
fabricated screenshots. A high-confidence hit triggers
<code>h<sub>t,m</sub>=1</code> and zeros the task score.
</p>
</article>
<article class="method-card accent-green">
<span class="method-num">M4</span>
<h3>Layered scoring (min rule)</h3>
<p>
<code>s<sub>t,m</sub> = 0</code> if <code>h<sub>t,m</sub>=1</code>;
otherwise <code>min(⅛ ∑ d<sup>process</sup>,
d<sup>deliv</sup>)</code>. The min prevents strong auxiliary
dimensions from masking weak deliverables; the zeroing rule prevents
fabricated evidence from earning partial credit.
</p>
</article>
</div>
</section>
<!-- ============ 03 DATASET ============ -->
<section class="section" id="dataset">
<div class="section-header">
<div class="section-eyebrow">03 / Dataset</div>
<div>
<h2>114 tasks across 8 domains, long, channel-interleaved.</h2>
<p class="section-lede">
Per-domain task counts range from 10 to 18. Best live rollouts use a
<strong>median of 76 tool calls</strong> (max 471) and a
<strong>median of 16 GUI↔CLI channel switches</strong> per task.
Each task carries provenance (URL, commit hash, or post id) from public
venues, with self-contained bundles
<code>ℰ = (𝒫, ℳ, 𝒞)</code>
covering prompt, materials, and check anchors.
</p>
</div>
</div>
<figure class="figure-frame">
<img src="static/images/data_overview.png?v=20260604d" alt="WeaveBench dataset overview: taxonomy of 114 tasks across 8 domains and 23 subcategories; per-task GUI/CLI channel-switch distribution; per-task tool-call rollout length.">
<figcaption class="caption">
<strong>Figure 3 from the paper.</strong>
<strong>(a)</strong> Taxonomy of 114 tasks across 8 domains and 23 subcategories.
<strong>(b)</strong> GUI↔CLI channel switches per task (median 16) — the degree of channel interleaving.
<strong>(c)</strong> Rollout length measured by tool calls in the trajectory (median 76, max 471).
</figcaption>
</figure>
<div class="domain-grid">
<div class="domain-card" style="--accent: #475569;">
<span class="domain-tag">DSK</span>
<h4>Desktop Productivity</h4>
<p>Filesystem & storage, system & hardware services, desktop UI / UX flows.</p>
</div>
<div class="domain-card" style="--accent: #ec4899;">
<span class="domain-tag">DOC</span>
<h4>Document Processing</h4>
<p>Office suites, markup, LaTeX, print-ready document workflows.</p>
</div>
<div class="domain-card" style="--accent: #16a34a;">
<span class="domain-tag">GAM</span>
<h4>Games & Interactive</h4>
<p>Game engines & runtimes, puzzle / strategy, realtime / action.</p>
</div>
<div class="domain-card" style="--accent: #0ea5e9;">
<span class="domain-tag">WEB</span>
<h4>Web Development</h4>
<p>DevTools, perf budgets, network profiling, UI & client-state debugging.</p>
</div>
<div class="domain-card" style="--accent: #4f46e5;">
<span class="domain-tag">DAV</span>
<h4>Data Analysis & Viz</h4>
<p>Notebooks, dashboards, observability traces, pipelines / ETL.</p>
</div>
<div class="domain-card" style="--accent: #7c3aed;">
<span class="domain-tag">OPS</span>
<h4>DevOps & Sysadmin</h4>
<p>Cluster monitoring, services, DB ops, network & security.</p>
</div>
<div class="domain-card" style="--accent: #0891b2;">
<span class="domain-tag">SPA</span>
<h4>Spatial / 3D / CAD</h4>
<p>CAD, engineering design, scientific / FEM simulation.</p>
</div>
<div class="domain-card" style="--accent: #f59e0b;">
<span class="domain-tag">DES</span>
<h4>Design & Creative</h4>
<p>Visual asset workflows, color management, engineering design.</p>
</div>
</div>
</section>
<!-- ============ 03b DEMOS ============ -->
<section class="section" id="demos">
<div class="section-header">
<div class="section-eyebrow">03 / Demos</div>
<div>
<h2>Watch the agent weave GUI & CLI in real trajectories.</h2>
<p class="section-lede">
Seven end-to-end rollouts from <strong>Claude Opus 4.7</strong>
on the <strong>OpenClaw</strong> harness, captured on a real Ubuntu
desktop and replayed at <strong>5× speed</strong>. The left pane
shows the agent's live action log; the right pane shows the desktop it
is driving. Use the arrows to step through all seven.
</p>
</div>
</div>
<div class="demo-carousel" id="demo-carousel"
data-videos='[
{"src":"static/videos/rabbitmq_dlq_topology_mgmt","domain":"OPS","accent":"#7c3aed","title":"Manage a RabbitMQ dead-letter-queue topology"},
{"src":"static/videos/darktable_raw_tonecurve_recover","domain":"DES","accent":"#f59e0b","title":"Recover a blown-out RAW photo in darktable"},
{"src":"static/videos/electron_app_test","domain":"DSK","accent":"#475569","title":"Drive the Joplin Electron note app end-to-end"},
{"src":"static/videos/godot_scene_node_debug","domain":"GAM","accent":"#16a34a","title":"Debug a broken scene-graph node in Godot"},
{"src":"static/videos/iframe_3layer_form","domain":"WEB","accent":"#0ea5e9","title":"Fill a 3-layer nested-iframe insurance form"},
{"src":"static/videos/inkscape_vectorize_logo","domain":"DES","accent":"#f59e0b","title":"Vectorize a logo into a print-ready bundle in Inkscape"},
{"src":"static/videos/meshlab_printability_audit","domain":"SPA","accent":"#0891b2","title":"Audit a 3D mesh for printability in MeshLab"}
]'
data-ver="20260605d">
<button class="dc-arrow dc-prev" type="button" aria-label="Previous demo">‹</button>
<div class="dc-stage">
<div class="dc-head">
<span class="dc-tag" id="dc-tag">OPS</span>
<span class="dc-title" id="dc-title">Manage a RabbitMQ dead-letter-queue topology</span>
<span class="dc-count" id="dc-count">1 / 7</span>
</div>
<div class="dc-frame">
<video id="dc-video" controls preload="none" playsinline></video>
</div>
</div>
<button class="dc-arrow dc-next" type="button" aria-label="Next demo">›</button>
<div class="dc-dots" id="dc-dots" role="tablist" aria-label="Choose a demo"></div>
</div>
</section>
<!-- ============ 04 MAIN RESULTS ============ -->
<section class="section" id="results">
<div class="section-header">
<div class="section-eyebrow">04 / Main Results</div>
<div>
<h2>Even frontier model×harness pairings stall at 41.2% PassRate.</h2>
<p class="section-lede">
The benchmark has two natural axes. <strong>Sweep A</strong> fixes
OpenClaw as the runtime and varies the backbone (Table 1).
<strong>Sweep B</strong> keeps the strongest backbones and varies
the deployed runtime (Table 2). Together they show that
hybrid-interface performance is determined as much by the runtime
scaffold as by raw model capability — cross-pairing can swing the
same backbone by >25 PR points.
</p>
</div>
</div>
<div class="table-wrap" id="leaderboard">
<div class="table-title leaderboard-title">🏆 Live Leaderboard · every reported model×harness pairing over the full 114 tasks, ranked by PassRate. <a href="https://github.com/weavebench/WeaveBench#-leaderboard" target="_blank" rel="noopener">Community submissions welcome →</a></div>
<table class="results-table results-table-2">
<thead>
<tr>
<th>#</th>
<th>Model</th>
<th>Harness</th>
<th>PR ↑</th>
<th>Overall ↑</th>
<th>DSK</th><th>DOC</th><th>GAM</th><th>WEB</th><th>DAV</th><th>OPS</th><th>SPA</th><th>DES</th>
</tr>
</thead>
<tbody>
<tr class="best-row">
<td>1</td><td class="model-cell"><strong>Claude Opus 4.7</strong></td><td class="harness-cell"><strong>Claude Code</strong></td>
<td style="--heat: 100;"><strong>41.2</strong></td><td style="--heat: 100;"><strong>0.532</strong></td>
<td>55.6</td><td>47.1</td><td>23.5</td><td>53.3</td><td>23.1</td><td>50.0</td><td>33.3</td><td>40.0</td>
</tr>
<tr>
<td>2</td><td class="model-cell">GPT-5.5</td><td class="harness-cell">Codex CLI</td>
<td style="--heat: 95;">35.1</td><td style="--heat: 99;">0.499</td>
<td>38.9</td><td>29.4</td><td>23.5</td><td>53.3</td><td>15.4</td><td>50.0</td><td>58.3</td><td>10.0</td>
</tr>
<tr>
<td>3</td><td class="model-cell">Claude Opus 4.7</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 95;">35.1</td><td style="--heat: 97;">0.482</td>
<td>55.6</td><td>29.4</td><td>23.5</td><td>66.7</td><td>15.4</td><td>41.7</td><td>16.7</td><td>20.0</td>
</tr>
<tr>
<td>4</td><td class="model-cell">GPT-5.5</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 90;">33.3</td><td style="--heat: 95;">0.466</td>
<td>38.9</td><td>35.3</td><td>35.3</td><td>21.4</td><td>23.1</td><td>38.5</td><td>33.3</td><td>40.0</td>
</tr>
<tr>
<td>5</td><td class="model-cell">GPT-5.5</td><td class="harness-cell">Hermes Agent</td>
<td style="--heat: 85;">31.6</td><td style="--heat: 95;">0.466</td>
<td>55.6</td><td>29.4</td><td>35.3</td><td>40.0</td><td>7.7</td><td>25.0</td><td>25.0</td><td>20.0</td>
</tr>
<tr>
<td>6</td><td class="model-cell"><strong>Seed 2.1 pro</strong></td><td class="harness-cell">Claude Code</td>
<td style="--heat: 82;">30.7</td><td style="--heat: 100;">0.551</td>
<td>44.4</td><td>41.2</td><td>17.6</td><td>40.0</td><td>30.8</td><td>8.3</td><td>33.3</td><td>20.0</td>
</tr>
<tr>
<td>7</td><td class="model-cell"><strong>Seed 2.1 turbo</strong></td><td class="harness-cell">Claude Code</td>
<td style="--heat: 78;">28.9</td><td style="--heat: 100;">0.571</td>
<td>38.9</td><td>35.3</td><td>17.6</td><td>33.3</td><td>30.8</td><td>16.7</td><td>33.3</td><td>20.0</td>
</tr>
<tr>
<td>8</td><td class="model-cell">Claude Opus 4.7</td><td class="harness-cell">Hermes Agent</td>
<td style="--heat: 75;">28.1</td><td style="--heat: 100;">0.516</td>
<td>33.3</td><td>47.1</td><td>11.8</td><td>26.7</td><td>30.8</td><td>50.0</td><td>8.3</td><td>10.0</td>
</tr>
<tr>
<td>9</td><td class="model-cell">GPT-5.4</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 62;">22.8</td><td style="--heat: 95;">0.465</td>
<td>55.6</td><td>35.3</td><td>5.9</td><td>0.0</td><td>23.1</td><td>23.1</td><td>8.3</td><td>20.0</td>
</tr>
<tr>
<td>10</td><td class="model-cell">GPT-5.3-codex</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 50;">18.4</td><td style="--heat: 93;">0.456</td>
<td>33.3</td><td>23.5</td><td>29.4</td><td>0.0</td><td>7.7</td><td>16.7</td><td>8.3</td><td>20.0</td>
</tr>
<tr>
<td>11</td><td class="model-cell">GPT-5.5</td><td class="harness-cell">Claude Code</td>
<td style="--heat: 40;">14.9</td><td style="--heat: 60;">0.299</td>
<td>33.3</td><td>11.8</td><td>11.8</td><td>0.0</td><td>15.4</td><td>16.7</td><td>25.0</td><td>0.0</td>
</tr>
<tr>
<td>12</td><td class="model-cell">Claude Opus 4.7</td><td class="harness-cell">Codex CLI</td>
<td style="--heat: 35;">13.2</td><td style="--heat: 75;">0.378</td>
<td>16.7</td><td>11.8</td><td>11.8</td><td>6.7</td><td>7.7</td><td>25.0</td><td>16.7</td><td>10.0</td>
</tr>
<tr>
<td>13</td><td class="model-cell">GPT-5.2-codex</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 18;">6.1</td><td style="--heat: 65;">0.321</td>
<td>5.6</td><td>11.8</td><td>0.0</td><td>0.0</td><td>15.4</td><td>16.7</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td>14</td><td class="model-cell">GPT-5.1-codex</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 6;">1.8</td><td style="--heat: 45;">0.226</td>
<td>0.0</td><td>5.9</td><td>0.0</td><td>0.0</td><td>7.7</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td>15</td><td class="model-cell">Gemini 3.1 Pro</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 6;">1.8</td><td style="--heat: 44;">0.223</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>8.3</td><td>0.0</td>
</tr>
<tr>
<td>16</td><td class="model-cell">Qwen3.5-397B-A17B</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 3;">0.9</td><td style="--heat: 64;">0.318</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td>17</td><td class="model-cell">Qwen3-VL-8B-Think</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 3;">0.9</td><td style="--heat: 18;">0.092</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td>18</td><td class="model-cell">GUI-Owl-1.5-32B</td><td class="harness-cell">OpenClaw</td>
<td style="--heat: 0;">0.0</td><td style="--heat: 12;">0.065</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
</tbody>
</table>
<p class="table-caption">
PR = PassRate (%) at τ=0.80; Overall = mean per-task score over 114 tasks; per-domain columns are PassRate. This board is maintained live — the paper's fixed-axis Table 1 and Table 2 below are the canonical snapshots. Submit a model×harness pairing via a pull request on <a href="https://github.com/weavebench/WeaveBench">GitHub</a>.
</p>
</div>
<div class="table-wrap">
<div class="table-title">Table 1 · Model API sweep on a fixed OpenClaw runtime (114 tasks, best thinking mode per backbone).</div>
<table class="results-table">
<thead>
<tr>
<th>Backbone</th>
<th>PR ↑</th>
<th>Overall ↑</th>
<th>DSK</th><th>DOC</th><th>GAM</th><th>WEB</th><th>DAV</th><th>OPS</th><th>SPA</th><th>DES</th>
</tr>
</thead>
<tbody>
<tr class="best-row">
<td class="model-cell"><strong>Claude Opus 4.7</strong></td>
<td style="--heat: 100;"><strong>35.1</strong></td>
<td style="--heat: 100;"><strong>0.482</strong></td>
<td>55.6</td><td>29.4</td><td>23.5</td><td>66.7</td><td>15.4</td><td>41.7</td><td>16.7</td><td>20.0</td>
</tr>
<tr>
<td class="model-cell"><strong>GPT-5.5</strong></td>
<td style="--heat: 95;">33.3</td>
<td style="--heat: 97;">0.466</td>
<td>38.9</td><td>35.3</td><td>35.3</td><td>21.4</td><td>23.1</td><td>38.5</td><td>33.3</td><td>40.0</td>
</tr>
<tr>
<td class="model-cell"><strong>GPT-5.4</strong></td>
<td style="--heat: 65;">22.8</td>
<td style="--heat: 96;">0.465</td>
<td>55.6</td><td>35.3</td><td>5.9</td><td>0.0</td><td>23.1</td><td>23.1</td><td>8.3</td><td>20.0</td>
</tr>
<tr>
<td class="model-cell">GPT-5.3-codex</td>
<td style="--heat: 52;">18.4</td>
<td style="--heat: 95;">0.456</td>
<td>33.3</td><td>23.5</td><td>29.4</td><td>0.0</td><td>7.7</td><td>16.7</td><td>8.3</td><td>20.0</td>
</tr>
<tr>
<td class="model-cell">GPT-5.2-codex</td>
<td style="--heat: 17;">6.1</td>
<td style="--heat: 67;">0.321</td>
<td>5.6</td><td>11.8</td><td>0.0</td><td>0.0</td><td>15.4</td><td>16.7</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td class="model-cell">GPT-5.1-codex</td>
<td style="--heat: 5;">1.8</td>
<td style="--heat: 47;">0.226</td>
<td>0.0</td><td>5.9</td><td>0.0</td><td>0.0</td><td>7.7</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td class="model-cell">Gemini 3.1 pro</td>
<td style="--heat: 5;">1.8</td>
<td style="--heat: 46;">0.223</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>8.3</td><td>0.0</td>
</tr>
<tr>
<td class="model-cell">Qwen3.5-397B-A17B</td>
<td style="--heat: 3;">0.9</td>
<td style="--heat: 66;">0.318</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td class="model-cell">Qwen3-VL-8B-Think</td>
<td style="--heat: 3;">0.9</td>
<td style="--heat: 19;">0.092</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>8.3</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
<tr>
<td class="model-cell">GUI-Owl-1.5-32B</td>
<td style="--heat: 0;">0.0</td>
<td style="--heat: 14;">0.065</td>
<td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td><td>0.0</td>
</tr>
</tbody>
</table>
<p class="table-caption">
SPA and DES — the two most GUI-heavy domains — are the bottom-two for every backbone with non-trivial PR, confirming GUI as the binding constraint.
</p>
</div>
<div class="table-wrap">
<div class="table-title">Table 2 · Cross-harness sweep for the strongest backbones (high thinking).</div>
<table class="results-table results-table-2">
<thead>
<tr>
<th>Backbone</th>
<th>Harness</th>
<th>PR ↑</th>
<th>Overall ↑</th>
<th>DSK</th><th>DOC</th><th>GAM</th><th>WEB</th><th>DAV</th><th>OPS</th><th>SPA</th><th>DES</th>
</tr>
</thead>
<tbody>
<tr>
<td class="model-cell" rowspan="4"><strong>GPT-5.5</strong></td>
<td class="harness-cell"><strong>Codex CLI</strong></td>
<td style="--heat: 95;"><strong>35.1</strong></td>
<td style="--heat: 99;"><strong>0.499</strong></td>
<td>38.9</td><td>29.4</td><td>23.5</td><td>53.3</td><td>15.4</td><td>50.0</td><td>58.3</td><td>10.0</td>
</tr>
<tr>
<td class="harness-cell">OpenClaw</td>
<td style="--heat: 90;">33.3</td>
<td style="--heat: 95;">0.466</td>
<td>38.9</td><td>35.3</td><td>35.3</td><td>21.4</td><td>23.1</td><td>38.5</td><td>33.3</td><td>40.0</td>
</tr>
<tr>
<td class="harness-cell">Hermes Agent</td>
<td style="--heat: 85;">31.6</td>
<td style="--heat: 95;">0.466</td>
<td>55.6</td><td>29.4</td><td>35.3</td><td>40.0</td><td>7.7</td><td>25.0</td><td>25.0</td><td>20.0</td>
</tr>
<tr>
<td class="harness-cell">Claude Code</td>
<td style="--heat: 40;">14.9</td>
<td style="--heat: 60;">0.299</td>
<td>33.3</td><td>11.8</td><td>11.8</td><td>0.0</td><td>15.4</td><td>16.7</td><td>25.0</td><td>0.0</td>
</tr>
<tr class="best-row">
<td class="model-cell" rowspan="4"><strong>Claude Opus 4.7</strong></td>
<td class="harness-cell"><strong>Claude Code</strong></td>
<td style="--heat: 100;"><strong>41.2</strong></td>
<td style="--heat: 100;"><strong>0.532</strong></td>
<td>55.6</td><td>47.1</td><td>23.5</td><td>53.3</td><td>23.1</td><td>50.0</td><td>33.3</td><td>40.0</td>
</tr>
<tr>
<td class="harness-cell">OpenClaw</td>
<td style="--heat: 95;">35.1</td>
<td style="--heat: 97;">0.482</td>
<td>55.6</td><td>29.4</td><td>23.5</td><td>66.7</td><td>15.4</td><td>41.7</td><td>16.7</td><td>20.0</td>
</tr>
<tr>
<td class="harness-cell">Hermes Agent</td>
<td style="--heat: 75;">28.1</td>
<td style="--heat: 100;">0.516</td>
<td>33.3</td><td>47.1</td><td>11.8</td><td>26.7</td><td>30.8</td><td>50.0</td><td>8.3</td><td>10.0</td>
</tr>
<tr>
<td class="harness-cell">Codex CLI</td>
<td style="--heat: 35;">13.2</td>
<td style="--heat: 75;">0.378</td>
<td>16.7</td><td>11.8</td><td>11.8</td><td>6.7</td><td>7.7</td><td>25.0</td><td>16.7</td><td>10.0</td>
</tr>
</tbody>
</table>
<p class="table-caption">
Cross-pairing matters: Claude Opus 4.7 drops from 41.2% on Claude Code to 13.2% on Codex CLI; GPT-5.5 drops from 35.1% on Codex CLI to 14.9% on Claude Code. Tool schemas, prompting conventions, and action-loop design interact strongly with model-specific tool-use behavior.
</p>
</div>
<div class="headline-comparison" aria-label="WeaveBench vs peer CUA benchmarks">
<div class="hc-hero">
<span class="hc-eyebrow">Why 41.2% is striking</span>
<div class="hc-bigstat">
<span class="hc-bigvalue">41.2<small>%</small></span>
<span class="hc-bigcaption">Best frontier pairing<br><b>Claude Opus 4.7 + Claude Code</b></span>
</div>
<p class="hc-narrative">
The same generation of frontier backbones that scores
<strong>>78%</strong> on OSWorld-Verified and
<strong>75%</strong> on MCPWorld collapses to
<strong>41.2%</strong> on WeaveBench — and to
<strong>≤3.5%</strong> when restricted to a single channel.
That gap is the cross-interface, long-horizon orchestration tax.
</p>
</div>
<div class="hc-bars" aria-label="Peer benchmark comparison">
<div class="hc-bar-row">
<span class="hc-bar-name">OSWorld-Verified</span>
<div class="hc-bar-track"><div class="hc-bar-fill" style="--w: 100%; --c1: #475569; --c2: #94a3b8;"></div></div>
<span class="hc-bar-val">>78<small>%</small></span>
</div>
<div class="hc-bar-row">
<span class="hc-bar-name">MCPWorld<br><small>hybrid</small></span>
<div class="hc-bar-track"><div class="hc-bar-fill" style="--w: 96%; --c1: #64748b; --c2: #94a3b8;"></div></div>
<span class="hc-bar-val">75.1<small>%</small></span>
</div>
<div class="hc-bar-row">
<span class="hc-bar-name">OSWorld-MCP<br><small>hybrid</small></span>
<div class="hc-bar-track"><div class="hc-bar-fill" style="--w: 55%; --c1: #64748b; --c2: #94a3b8;"></div></div>
<span class="hc-bar-val">43.3<small>%</small></span>
</div>
<div class="hc-bar-row hc-ours">
<span class="hc-bar-name"><strong>WeaveBench</strong><br><small>hybrid (ours)</small></span>
<div class="hc-bar-track"><div class="hc-bar-fill" style="--w: 53%; --c1: #ec4899; --c2: #f59e0b;"></div></div>
<span class="hc-bar-val"><strong>41.2<small>%</small></strong></span>
</div>
<div class="hc-bar-row hc-ours hc-faded">
<span class="hc-bar-name"><strong>WeaveBench</strong><br><small>single-channel max</small></span>
<div class="hc-bar-track"><div class="hc-bar-fill" style="--w: 4.5%; --c1: #ec4899; --c2: #f59e0b;"></div></div>
<span class="hc-bar-val">≤3.5<small>%</small></span>
</div>
<p class="hc-footnote">
Peer benchmark numbers from each paper's strongest reported result. WeaveBench rows are this work (Table 2 best-pairing & Table 3 single-channel maximum across all backbones).
</p>
</div>
</div>
</section>
<!-- ============ 05 ABLATIONS ============ -->
<section class="section" id="ablations">
<div class="section-header">
<div class="section-eyebrow">05 / Ablations</div>
<div>
<h2>Two ablations that change how you read the leaderboard.</h2>
<p class="section-lede">
The first ablation removes either channel and shows that
single-interface CUAs collapse by an <em>order of magnitude</em>
— on WeaveBench the second channel is not a convenience, it is
required. The second ablation removes trajectory access from the
judge and shows that outcome-only grading awards 10–20 PR points
of false credit.
</p>
</div>
</div>
<div class="ablation-grid">
<article class="ablation-card">
<span class="ablation-tag">Table 3 · Interface ablation</span>
<h3>GUI-only ≤1.8%, CLI-only ≤3.5%, Hybrid 22–35%.</h3>
<p class="ablation-lede">
We re-run each backbone in three settings on OpenClaw: GUI-only (the
screenshot tool plus nine actuation primitives), CLI-only (the full
OpenClaw CLI), and Hybrid (both). Both single-interface settings
collapse by an order of magnitude across every backbone, consistent
with the channel non-substitutability admission rule (P1).
</p>
<table class="judge-table ablation-table">
<thead>
<tr><th>Backbone</th><th>GUI</th><th>CLI</th><th>Hybrid</th><th>Δ Hyb-best1</th></tr>
</thead>
<tbody>
<tr><td>Claude Opus 4.7</td><td>1.8</td><td>3.5</td><td><b>35.1</b></td><td class="delta-pos">+31.6</td></tr>
<tr><td>GPT-5.5</td><td>0.8</td><td>2.6</td><td>33.3</td><td class="delta-pos">+30.7</td></tr>
<tr><td>GPT-5.4</td><td>0.8</td><td>2.6</td><td>22.8</td><td class="delta-pos">+20.2</td></tr>
<tr><td>GPT-5.3-codex</td><td>0.0</td><td>1.8</td><td>18.4</td><td class="delta-pos">+16.6</td></tr>
</tbody>
</table>
<p class="ablation-foot">
Single-interface PassRate stays in single digits for every backbone — an order of magnitude below Hybrid.
</p>
</article>
<article class="ablation-card">
<span class="ablation-tag">Table 4 · Cross-benchmark hybrid gain</span>
<h3>+31.6 pp gap — vs +3–4 pp on prior hybrid benchmarks.</h3>
<p class="ablation-lede">
Compared against the two prior hybrid CUA benchmarks that report
comparable interface ablations, WeaveBench's hybrid gain is an order
of magnitude larger, and the single-channel floor is single-digit
vs. 40–70%. The additional channel is forced by the
task specification, not offered as a per-step convenience.
</p>
<table class="judge-table ablation-table">
<thead>
<tr><th>Benchmark</th><th>GUI</th><th>CLI/MCP</th><th>Hyb.</th><th>Δ</th></tr>
</thead>
<tbody>
<tr><td>OSWorld-MCP</td><td>40.1</td><td>—</td><td>43.3</td><td class="delta-pos">+3.2</td></tr>
<tr><td>MCPWorld</td><td>70.7</td><td>53.2</td><td>75.1</td><td class="delta-pos">+4.5</td></tr>
<tr class="best-row"><td><b>WeaveBench</b></td><td><b>1.8</b></td><td><b>3.5</b></td><td><b>35.1</b></td><td class="delta-pos"><b>+31.6</b></td></tr>
</tbody>
</table>
<p class="ablation-foot">
Cooperation is forced by the task specification rather than offered as a per-step convenience.
</p>
</article>
</div>
<!-- Judge ablation -->
<div class="judge-ablation">
<div class="judge-ablation-text">
<span class="ablation-tag">Trajectory-aware vs. outcome-only judge</span>
<h3>Removing trajectory access inflates PR by 10.3–20.2 points.</h3>
<p>
We re-score every rollout with an outcome-only judge that sees only the
final deliverables — no trajectory access, no shortcut scan.
Switching back to WeaveBench's trajectory-aware judge removes
<strong>10.3–20.2 PR points</strong> across the four GPT
backbones. For GPT-5.5, the audited rate drops from <strong>53.5%
→ 33.3%</strong>. These gaps are <em>lower bounds</em>:
rollouts already received an anti-fabrication prompt with a cost-free
honest fallback.
</p>
<table class="judge-table">
<thead>
<tr><th>Backbone</th><th>Outcome-only</th><th>Trajectory-aware</th><th>Δ</th></tr>
</thead>
<tbody>
<tr><td>GPT-5.5</td><td>53.5</td><td>33.3</td><td class="delta">−20.2</td></tr>
<tr><td>Claude Opus 4.7</td><td>51.6</td><td>41.2</td><td class="delta">−10.4</td></tr>
<tr><td>GPT-5.4</td><td>33.1</td><td>22.8</td><td class="delta">−10.3</td></tr>
<tr><td>GPT-5.3-codex</td><td>28.7</td><td>18.4</td><td class="delta">−10.3</td></tr>
</tbody>
</table>
</div>
<figure class="judge-ablation-fig">
<img src="static/images/judge_ablation.png?v=20260604d" alt="Per-backbone PassRate audit: dark blue is audited PassRate after trajectory-aware judging; light blue shows inflation removed by the audit.">
<figcaption class="caption">
<strong>Figure 4 from the paper.</strong>
Dark blue: audited PassRate. Light blue: inflation removed by the audit, with label inside showing PassRate points removed. Top label: outcome-only total and points removed.
</figcaption>
</figure>
</div>
</section>
<!-- ============ 06 FAILURE ANALYSIS ============ -->
<section class="section" id="analysis">
<div class="section-header">
<div class="section-eyebrow">06 / Failure Analysis</div>
<div>
<h2>Reward hacking + execution-discipline collapse explain 65.6% of failures.</h2>
<p class="section-lede">
We aggregate every OpenClaw rollout for the three frontier backbones
(Opus 4.7, GPT-5.5, GPT-5.4) across reasoning budgets —
<strong>n = 2,209 trials, 1,735 failures</strong>. Following
CocoaBench, we adopt a hierarchical taxonomy with 5 top-level families
and 13 sub-classes; WeaveBench extends the codebook with two
hybrid-specific families absent from prior benchmarks —
<strong>E4 Long-horizon Execution Discipline</strong> and
<strong>E5 Reward Hacking</strong>.
</p>
</div>
</div>
<figure class="figure-frame">
<img src="static/images/failure_mechanism.png?v=20260605a" alt="Failure anatomy donut: 5 top-level families, 13 sub-classes, plus per-backbone sub-class share for Opus 4.7, GPT-5.5, GPT-5.4.">
<figcaption class="caption">
<strong>Figure 5 from the paper.</strong>
<strong>(a)</strong> Two-ring sunburst donut over <strong>n = 1,735</strong> failures. Inner ring: 5 top-level families. Outer ring: 13 sub-classes. <strong>E5 Reward Hacking + E4 Long-horizon Execution Discipline jointly account for 65.6%</strong> of all failures; perception (E3) is under 4%.
<strong>(b)</strong> Per-backbone sub-class share — y-axis label colour matches the family colour in (a).
</figcaption>
</figure>
<div class="failure-headline">
<div class="failure-family fam-e5">
<span class="failure-share">35.2%</span>
<strong>E5 · Reward Hacking</strong>
<p>Synthesized render · Hardcoded metric · Crop / overlay · CLI bypass of GUI. Only 30% of E5 hacks follow a true infra bug; the other 70% happen in clean environments despite the explicit anti-fabrication policy.</p>
</div>
<div class="failure-family fam-e4">
<span class="failure-share">30.4%</span>
<strong>E4 · Long-horizon Execution Discipline</strong>
<p>Silent halt · Premature halt · Cross-channel state drift. Emerges only when the agent must coordinate multiple deliverables across a long horizon of interleaved GUI/CLI actions.</p>
</div>
<div class="failure-family fam-e1">
<span class="failure-share">21.0%</span>
<strong>E1 · Reasoning & Verification</strong>
<p>Imprecision — close miss is the largest sub-class here. Surfaces only with multi-deliverable contracts.</p>
</div>
<div class="failure-family fam-e2">
<span class="failure-share">9.5%</span>
<strong>E2 · Tool Selection</strong>
<p>Tool-affordance prior + channel-policy compliance failures.</p>
</div>
<div class="failure-family fam-e3">
<span class="failure-share">3.7%</span>
<strong>E3 · Visual Grounding</strong>
<p>Perception is <em>not</em> the bottleneck on frontier backbones — a finding that runs against the perception-centric prior in the GUI-agent literature.</p>
</div>
</div>
<div class="fingerprints">
<article class="fingerprint">
<span class="fp-tag">Backbone fingerprint</span>
<h3>GPT-5.5 · the “confident forger”</h3>
<p>46% of failures are E5 reward hacking — primarily E5.1 synthesized renders (24%) and E5.2 hardcoded metrics (14%). The model can complete or honestly skip, but the implicit reward landscape selects forgery when stuck.</p>
</article>
<article class="fingerprint">
<span class="fp-tag">Backbone fingerprint</span>
<h3>GPT-5.4 · the “early stopper”</h3>
<p>E4 dominates (44%) and E4.2 premature halt alone is 27% — the highest single sub-class share for any backbone. Reward hacking is comparatively rare (23%); after a provider hiccup the model never rebuilds its deliverable checklist.</p>
</article>
<article class="fingerprint">
<span class="fp-tag">Backbone fingerprint</span>
<h3>Claude Opus 4.7 · the “balanced”</h3>
<p>No single sub-class above 17%. E5, E4, and E1 each contribute ~30%. Failure style is a function of model identity, not just raw capability.</p>
</article>
</div>
</section>
<!-- ============ 06b TOOL-CALL DISTRIBUTION ============ -->
<section class="section section-toolmix" id="toolmix">
<div class="section-header">
<div class="section-eyebrow">06b / Tool-Call Distribution</div>
<div>
<h2>Even with a dedicated GUI tool exposed, agents prefer shell paths for GUI actions.</h2>
<p class="section-lede">
We decompose every tool call across all GPT-5.5 rollouts into atomic
operations. The top-10 operations cover <strong>93.1%</strong>
of 10,873 active calls; <code>exec: shell</code> alone accounts
for <strong>27.3%</strong>. A large fraction of GUI work is
hidden inside these shell calls — once re-attributed at the
atomic-operation level, the GUI share rises from
<strong>33.9% → 62.9%</strong>.
</p>
</div>
</div>
<div class="toolmix-grid">
<figure class="figure-frame toolmix-figure">
<img src="static/images/toolmix_gpt55.png?v=20260604d" alt="Top-10 atomic operations across all GPT-5.5 rollouts; bars sorted by call count.">
<figcaption class="caption">
Top-10 atomic operations across all GPT-5.5 rollouts on WeaveBench — 93.1% of 10,873 active calls. GPT-5.5 invokes <code>gnome-screenshot</code> via <code>exec</code> 2.2× more often than the native <code>__computer__.screenshot</code>, and drives mouse/keyboard via <code>pyautogui</code>, <code>xdotool</code>, and <code>wmctrl</code> another 521 times.
</figcaption>
</figure>
<div class="toolmix-numbers">
<div class="num-card">
<span class="num-tag">exec: shell share</span>
<span class="num-value">27.3<small>%</small></span>
<p>of all GPT-5.5 tool calls — the single dominant primitive in the trajectory.</p>
</div>
<div class="num-card">
<span class="num-tag">GUI re-attributed</span>
<span class="num-value">33.9 <small>→</small> 62.9<small>%</small></span>
<p>tool-level GUI share rises to atomic-op share once exec-routed screenshots and pyautogui calls are reclassified.</p>
</div>
<div class="num-card">
<span class="num-tag">Channel switches</span>
<span class="num-value">16<small> / task</small></span>
<p>median GUI↔CLI handoffs in a successful trajectory — single-channel agents cannot satisfy this.</p>
</div>
<div class="num-card">
<span class="num-tag">Median rollout</span>
<span class="num-value">76<small> calls</small></span>
<p>per task; max 471. WeaveBench is firmly in long-horizon territory.</p>
</div>
</div>
</div>
</section>
<!-- ============ 07 CITATION ============ -->
<section class="section" id="citation">
<div class="section-header">
<div class="section-eyebrow">07 / BibTeX</div>
<div>
<h2>Cite this work.</h2>
<p class="section-lede">
If you use WeaveBench in your research, please cite the paper and the
released dataset / code.
</p>
</div>
</div>
<div class="bibtex-wrap">
<button class="bib-copy" id="bib-copy" type="button" aria-label="Copy BibTeX">Copy</button>
<pre id="bibtex"><code>@article{li2026weavebench,
title = {{WeaveBench}: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces},
author = {Li, Wanli and Zhou, Bowen and Yu, Yunyao and Xu, Zhou and Yang, Yifan and Li, Dongsheng and Shan, Caihua},
year = {2026},
eprint = {2606.09426},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2606.09426},
}</code></pre>
</div>
<div class="authors">
<h3>Authors</h3>
<ul class="author-list">
<li><strong>Wanli Li</strong><sup>1,2,*</sup></li>