Skip to content

benchmark.md

Latest commit

 

History

History
28 lines (21 loc) · 2.33 KB

benchmark.md

File metadata and controls

28 lines (21 loc) · 2.33 KB
Raw

Benchmark

Generated by packages/bench (6 repetitions per scenario). Both sides run on the same fixtures with the same injected page agent. Ours = the full shipped harness (whole-page perception + semantic fingerprints + widget adapters + post-condition verification + self-healing + data-driven batch). Baseline = a deliberate replica of the common viewport-agent architecture (viewport-only perception + index addressing + native controls only + fixed 1s waits + no verification + stop on first failure). Source: packages/bench/src/baseline.ts.

Results

Metric (higher is better) Browser Agent Viewport baseline Why
Custom-widget task success (antd-style select, DOM ground truth) 100% 0% baseline can only drive native <select>
Fake-success detection (toast shown, nothing persisted) 100% 0% baseline has no verification — it self-reports success
Real completion under flakiness (DOM ground truth) 100% 0% baseline stops at first transient failure
Off-screen field hit rate (bottom of a long form, no scrolling) 100% 0% baseline only perceives the viewport
Batch delivery accuracy (reported == actually persisted) 100% 80% ours: reports 8/10, DB has 8. baseline: reports 10/10, DB has 8

Takeaways

  1. Verification is the watershed. In the fake-success scenario (page pops a success toast but persists nothing), objective post-conditions like list_count_delta catch the lie on the spot. An agent without verification calls it success every time — "action performed ≠ job done", quantified.
  2. Widget adapters decide real-world usability. An antd-style dropdown (a div shell) is completely undriveable for the baseline; the adapter layer selects the option and reads the value back for confirmation.
  3. Self-healing buys resilience. With injected flakiness, the baseline dies on first failure; diagnose → smart wait → retry completes the task.
  4. Whole-page perception removes scroll dependence. Fields below the fold are simply present in the semantic graph.
  5. Batch reports stay honest. Reported successes are cross-checked against records actually present in the page — the numbers must reconcile.

Reproduce

pnpm --filter @browser-agent/fixtures build
pnpm bench            # HEADED=1 to watch it run