Skip to content

Commit 76ada5b

Browse files
docs: update README with current benchmark numbers
1 parent 209dcc0 commit 76ada5b

1 file changed

Lines changed: 9 additions & 9 deletions

File tree

README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
Python implementation of [GCF (Graph Compact Format)](https://gcformat.com/) — the most token-efficient wire format for LLMs. A drop-in alternative to JSON and TOON for any structured data.
99

10-
**79% fewer input tokens than JSON. 75% fewer output tokens. 52% smaller than TOON. 100% LLM comprehension at 500 symbols, where JSON fails at 66.7%.**
10+
**79% fewer input tokens than JSON. 75% fewer output tokens. 52% smaller than TOON. 100% LLM comprehension at 500 symbols, where JSON scores 76.9% and TOON scores 92.3%.**
1111

1212
Docs: [gcformat.com](https://gcformat.com/) · [Playground](https://gcformat.com/playground.html) · [GCF vs TOON](https://gcformat.com/guide/vs-toon.html)
1313

@@ -164,27 +164,27 @@ Works on dicts, lists, and primitives. Lists of uniform dicts get tabular rows.
164164

165165
## Comprehension Eval
166166

167-
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. Six structured extraction questions sent to an LLM:
167+
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM:
168168

169169
| Format | Accuracy | Tokens | vs JSON |
170170
|--------|----------|--------|---------|
171-
| **GCF** | **100%** (6/6) | **11,090** | **79% fewer** |
172-
| TOON | 100% (6/6) | 16,378 | 69% fewer |
173-
| JSON | 66.7% (4/6) | 53,341 | baseline |
171+
| **GCF** | **100%** (13/13) | **11,090** | **79% fewer** |
172+
| TOON | 92.3% (12/13) | 16,378 | 69% fewer |
173+
| JSON | 76.9% (10/13) | 53,341 | baseline |
174174

175-
JSON failed on counting tasks. GCF and TOON both achieved perfect accuracy. GCF does it in 32% fewer tokens.
175+
GCF is the only format with perfect accuracy at scale, at 32% fewer tokens than TOON.
176176

177177
## Token Efficiency (TOON's Own Benchmark)
178178

179179
Running [TOON's benchmark harness](https://github.com/blackwell-systems/toon/tree/gcf-comparison) with GCF inserted (their datasets, their tokenizer):
180180

181181
| Track | GCF | TOON | Result |
182182
|-------|-----|------|--------|
183-
| Mixed-structure (nested, semi-uniform) | 169,554 | 227,896 | **GCF 34% smaller** |
183+
| Mixed-structure (nested, semi-uniform) | 170,367 | 227,896 | **GCF 34% smaller** |
184184
| Flat-only (tabular) | 66,026 | 67,837 | **GCF 3% smaller** |
185-
| Semi-uniform event logs | 107,269 | 154,032 | **GCF 44% smaller** |
185+
| Semi-uniform event logs | 108,158 | 154,032 | **GCF 42% smaller** |
186186

187-
GCF wins on every dataset except deeply nested config (75 tokens on a 618-token payload). On semi-uniform data, GCF uses 44% fewer tokens than TOON.
187+
GCF wins all 6 datasets. On semi-uniform data (the most common real-world pattern), GCF uses 42% fewer tokens than TOON.
188188

189189
Reproducible: [blackwell-systems/toon@gcf-comparison](https://github.com/blackwell-systems/toon/tree/gcf-comparison)
190190

0 commit comments

Comments
 (0)