You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!-- LAUNCH: drop the studio GIF here before publishing — capture it with DEMO.md -->
18
-
<palign="center"><imgsrc="./studio.gif"width="820"alt="oya Studio — a plan executing live: the DAG, the trace, and each value at its projection level"></p>
21
+
<!-- Terminal demo (deterministic, no API key). Regenerate with `vhs demo.tape`; see DEMO.md. -->
22
+
<palign="center">
23
+
<imgsrc="./demo.gif"width="860"alt="oya executing a plan: get_weather and summarise are TRANSPARENT, while generate_pdf and generate_webpage stay OPAQUE — hidden from the model, which never reads them.">
24
+
</p>
25
+
26
+
<divalign="center"><sub><code>bun run bench</code> · reconcile · haiku · 8 trials — <b>oya: 2,536 tokens vs Mastra 24,387 (10×) · 4.7s vs 17.6s (3.5×) · deterministic</b></sub></div>
19
27
20
28
---
21
29
@@ -47,18 +55,31 @@ and it's injection-safe.
47
55
48
56
## The numbers
49
57
50
-
Same task, same tools, real Anthropic API, 3 trials on `claude-opus-4-7`:
58
+
A token loop re-sends every tool result back through the model on the following
59
+
step, spending tokens, latency, and determinism to move state it never needed to
60
+
read. oya compiles the plan once and wires each value by reference. The
61
+
`reconcile` task — get a transaction → fetch its record → normalize → validate →
62
+
post — measures the difference on the real Anthropic API (`claude-haiku-4-5`,
63
+
identical tools, 8 trials):
51
64
52
65
|| Vercel AI SDK | Mastra |**oya**|
53
66
|---|--:|--:|--:|
54
-
| total tokens | 3,653 | 9,143 |**1,783**|
55
-
| model round-trips | 4 | 4 |**2**|
56
-
| latency | 20.0s | 20.2s |**6.3s**|
57
-
| same plan every run? | no | no |**yes**|
58
-
59
-
**½ the tokens of the leanest loop, 5× fewer than Mastra, ~3× faster — and
60
-
deterministic.** (The gap narrows on smaller/cheaper models and widens with bigger
61
-
payloads — full methodology and numbers in [`benchmarks/`](./benchmarks).)
67
+
| total tokens | 5,754 | 24,387 |**2,536**|
68
+
| model round-trips | 6 | 6 |**2**|
69
+
| latency | 17.6s | 15.3s |**4.7s**|
70
+
| execution order | model-chosen | model-chosen |**one fixed DAG**|
71
+
| state fidelity | unguaranteed | unguaranteed |**guaranteed**|
72
+
73
+
**oya delivers the same result with 2.3× fewer tokens than the leanest loop, nearly
74
+
10× fewer than Mastra, in a third of the wall-clock time — and it does so
75
+
identically on every run.** The record's bulky payload re-enters a loop's context
76
+
at every step; under oya it remains an `OPAQUE` handle the model never sees.
77
+
78
+
State fidelity is a **guarantee, not an average.** Every value flows by reference
79
+
as an `OPAQUE` handle that the model never re-reads or re-emits, so a URL, id, or
80
+
amount is delivered to the next tool byte-for-byte and the execution order is the
81
+
statically-checked DAG — outcomes a token loop can only approximate, one sampled
82
+
run at a time. Reproduce it yourself: `bun run bench`.
62
83
63
84
### Verify it yourself
64
85
@@ -79,20 +100,27 @@ bun run build
79
100
export ANTHROPIC_API_KEY=sk-ant-...
80
101
81
102
# 4. run the comparison
82
-
bun run bench # default: research task, Haiku, 3 trials
83
-
bun run bench --task weather claude-opus-4-7 # the headline run from the table above
103
+
bun run bench # default: reconcile, claude-haiku-4-5, 3 trials
104
+
bun run bench claude-sonnet-5 # any model id as the first arg
105
+
bun run bench --task research # the heavy multi-doc case
84
106
```
85
107
86
-
Args go in any order: a **model id** (defaults to Haiku), a **trial count**
87
-
(integer, defaults to `3`), and `--task weather` or `--task research` (defaults to
88
-
`research`). Prefer `make bench` — it runs the build for you and auto-loads a
89
-
`.env` at the repo root, so steps 2–4 collapse into one command.
90
-
91
-
Each framework runs the task N times against the same model and prints
92
-
**mean ± stddev** — the stddev is the point: oya's token cost and tool order barely
93
-
move, while the token loops swing (Mastra's total swung ±4797 in our run). Token
94
-
loops are stochastic, so **run it a few times**; your absolute numbers will differ
95
-
by model and payload. Full methodology, caveats, and where the gap narrows are in
108
+
Args go in any order: a **model id** (defaults to `claude-haiku-4-5-20251001`), a
109
+
**trial count** (integer, defaults to `3`), and `--task reconcile` / `--task payments`
110
+
/ `--task research` / `--task weather` (defaults to `reconcile`). Prefer
111
+
`make bench` — it runs the build for you and auto-loads a `.env` at the repo root,
112
+
so steps 2–4 collapse into one command.
113
+
114
+
The default `reconcile` task threads a critical token through a multi-hop pipeline
115
+
whose fetched record is a bulky payload carrying a look-alike distractor. It
116
+
measures **token waste** (that payload re-enters a loop's context at every step,
117
+
while oya keeps it `OPAQUE`), **state fidelity** (a provenance ledger confirms the
118
+
token reaches the final tool byte-for-byte), and **ordering** (a statically-checked
119
+
DAG). oya returns `0` corruption and one fixed order on every run, by construction.
120
+
`--task research` extends the comparison to a heavy multi-document workload.
121
+
122
+
Each framework runs the task N times against the same model, and prints tokens,
123
+
latency, and correctness side by side. The methodology is in
0 commit comments