You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+5-3Lines changed: 5 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -164,7 +164,7 @@ Works on dicts, lists, and primitives. Lists of uniform dicts get tabular rows.
164
164
165
165
## Comprehension Eval
166
166
167
-
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM:
167
+
Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM with zero format instructions:
168
168
169
169
| Format | Accuracy | Tokens | vs JSON |
170
170
|--------|----------|--------|---------|
@@ -174,19 +174,21 @@ Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 str
174
174
175
175
GCF is the only format with perfect accuracy at scale, at 32% fewer tokens than TOON.
176
176
177
+
Reproduce: `git clone https://github.com/blackwell-systems/gcf-go && cd gcf-go/eval && GOWORK=off go test -run TestComprehension -v -timeout 0`
178
+
177
179
## Token Efficiency (TOON's Own Benchmark)
178
180
179
181
Running [TOON's benchmark harness](https://github.com/blackwell-systems/toon/tree/gcf-comparison) with GCF inserted (their datasets, their tokenizer):
0 commit comments