Skip to content

Commit b51e531

Browse files
docs: standardize eval sections with reproduce commands
1 parent 76ada5b commit b51e531

1 file changed

Lines changed: 5 additions & 3 deletions

File tree

README.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -164,7 +164,7 @@ Works on dicts, lists, and primitives. Lists of uniform dicts get tabular rows.
164164

165165
## Comprehension Eval
166166

167-
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM:
167+
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM with zero format instructions:
168168

169169
| Format | Accuracy | Tokens | vs JSON |
170170
|--------|----------|--------|---------|
@@ -174,19 +174,21 @@ Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 str
174174

175175
GCF is the only format with perfect accuracy at scale, at 32% fewer tokens than TOON.
176176

177+
Reproduce: `git clone https://github.com/blackwell-systems/gcf-go && cd gcf-go/eval && GOWORK=off go test -run TestComprehension -v -timeout 0`
178+
177179
## Token Efficiency (TOON's Own Benchmark)
178180

179181
Running [TOON's benchmark harness](https://github.com/blackwell-systems/toon/tree/gcf-comparison) with GCF inserted (their datasets, their tokenizer):
180182

181183
| Track | GCF | TOON | Result |
182184
|-------|-----|------|--------|
183185
| Mixed-structure (nested, semi-uniform) | 170,367 | 227,896 | **GCF 34% smaller** |
184-
| Flat-only (tabular) | 66,026 | 67,837 | **GCF 3% smaller** |
186+
| Flat-only (tabular) | 66,029 | 67,837 | **GCF 3% smaller** |
185187
| Semi-uniform event logs | 108,158 | 154,032 | **GCF 42% smaller** |
186188

187189
GCF wins all 6 datasets. On semi-uniform data (the most common real-world pattern), GCF uses 42% fewer tokens than TOON.
188190

189-
Reproducible: [blackwell-systems/toon@gcf-comparison](https://github.com/blackwell-systems/toon/tree/gcf-comparison)
191+
Reproduce: `git clone https://github.com/blackwell-systems/toon && cd toon && git checkout gcf-comparison && cd benchmarks && pnpm install && pnpm benchmark:tokens`
190192

191193
## Links
192194

0 commit comments

Comments
 (0)