Why not

Author: gunta

README.md

@@ -1,117 +1,156 @@
 # The Format Tax
 
-**You're paying it right now. On every LLM call. On every streamed response. On every config file.**
+Benchmark suite, paper draft, and recommendation tool for serialization choices in human-AI systems.
 
-> The definitive, open-source guide to data serialization for human-AI systems. Benchmarks, decision framework, and the paper that proves JSON is overcharging you.
-
-**[formattax.dev](https://formattax.dev)** | [Read the Paper](docs/paper.md) | [Decision Tree](docs/decision-tree.md) | [Benchmarks](docs/benchmarks.md)
+**[formattax.dev](https://formattax.dev)** | [Paper draft](docs/paper.md) | [Recommendation policy](docs/decision-tree.md) | [Benchmarks](docs/benchmarks.md) | [Schemas](docs/schemas.md)
 
 ---
 
-## The Tax You Didn't Know You Were Paying
+## Thesis
 
-Every time you feed JSON to an LLM, you burn **30-72% of your token budget** on curly braces, repeated keys, and quotation marks. Every time you stream JSON to a UI, your users wait for a closing `}` that hasn't arrived yet. Every time you pick "just use JSON" for a config file, you lose the ability to document *why*.
+Serialization is not just an implementation detail.
 
-This isn't a style preference. It's a **tax** -- measured in dollars, latency, accuracy, and carbon.
+The same system may cross several boundaries with incompatible needs:
 
-## What's Inside
+- model-facing input
+- model-facing output
+- progressive UI streaming
+- human-maintained config
+- inter-service transport
+- storage
 
-```
-the-format-tax/
-├── README.md
-├── docs/
-│   ├── paper.md             # "The Format Tax" whitepaper (arXiv target)
-│   ├── prd.md               # Product Requirements Document
-│   ├── benchmarks.md        # Benchmark methodology & code
-│   ├── schemas.md           # Schema & validation ecosystem
-│   └── decision-tree.md     # Interactive decision framework
-├── benchmarks/              # Reproducible benchmark code
-├── site/                    # formattax.dev (Astro + Tailwind)
-└── LICENSE
-```
+Using one format for all of them imposes a *Format Tax*: unnecessary structural tokens, weaker streaming behavior, poorer ergonomics, or a less reliable output path than the stage actually requires.
 
-## The Tax Bill
+## What This Repo Is
 
-| What You're Doing | JSON's Tax | Best Alternative | Savings |
-|---|---|---|---|
-| Feeding tabular data to an LLM | 66,396 tokens / 1k records | ISON: 18,600 tokens | **72% fewer tokens** |
-| LLM retrieves a value from your data | 52.3% accuracy | Markdown-KV: 60.7% | **+8.4 pp** |
-| Streaming AI responses to UI | Waits for closing `}` | YAML (json-render): renders at every prefix | **3x faster first paint** |
-| Config file for your app | No comments, no docs | Markdown+Frontmatter | **Self-documenting** |
-| Backend service-to-service | 7,045 ns/op | FlatBuffers: 711 ns/op | **10x faster** |
-| Carbon footprint at scale | EES 0.926 | TOON: EES 0.980 | **6% less CO2 per call** |
+This repository contains three related artifacts:
 
-## The Thesis
+1. **A paper draft**
+   An academic-facing document that stays conservative about evidence and separates local results from external claims.
+2. **A benchmark suite**
+   Canonical datasets, question sets, encoders, and runner tracks for measuring format behavior.
+3. **A homepage and recommendation engine**
+   A practitioner-facing tool that returns a primary format, fallback, confidence, and schema/parser pairing.
 
-**There is no single best format.** The right answer depends on which stage of the pipeline you're at:
+## Evidence Policy
 
-```
-Storage -> Transport -> Validation -> Prompt Injection -> Generation -> Integration
-   |          |           |              |                |            |
-  JSON    Protobuf    Zod/Pydantic   MD-KV/TOON     JSON+constrained  JSON
-  Parquet  FlatBuf    JSON Schema      YAML            decoding      Protobuf
-                      Effect Schema   MD+Frontmatter
-```
+Every claim in the project should fit one of these buckets:
 
-Every stage has a different optimal format. Using JSON for all of them is the Format Tax.
+- **Locally measured** — produced by this repository’s benchmark code
+- **Externally reported** — from papers, official project benchmarks, or official docs
+- **Open hypothesis** — worth testing, not yet proven here
 
-## The Master Rule
+This matters because the public serialization discourse is noisy:
 
-```
-INPUT:    Markdown-KV / TOON family  (accuracy + tokens)
-OUTPUT:   JSON + constrained decoding (validity)
-STREAM:   YAML wire format           (prefix-valid)
-CONFIG:   YAML (complex) / TOML (simple)
-BACKEND:  Protobuf / FlatBuffers     (speed)
-VALIDATE: Zod / Pydantic / JSON Schema (stage-appropriate)
-```
+- some results are academic
+- some are official format-project benchmarks
+- some are independent reproductions
+- some are placeholder numbers used for site development
+
+The project is being rewritten to keep those apart.
+
+## Current Benchmark Coverage
+
+Implemented or in-progress runner tracks:
+
+- Token efficiency
+- Retrieval accuracy
+- Streaming readiness
+- Generation quality
+- Schema guidance
+
+Planned but not yet implemented as a local runner track:
+
+- Binary throughput / transport benchmarking
+
+## Recommendation Layer
+
+The homepage is intentionally more operational than the paper.
+
+It answers:
+
+- what boundary am I at?
+- what shape is the data?
+- what is the main optimization target?
+- what hard constraints apply?
+
+It then returns:
+
+- a primary recommendation
+- a fallback
+- confidence
+- evidence basis
+- schema pairing
+- parser pairing if relevant
 
-## Formats Covered
+The homepage is allowed to be decisive. The paper is not allowed to smuggle those decisions in as settled research results.
 
-- **20+ text formats** -- JSON, YAML, TOML, XML, CSV, JSONL, Markdown-KV, MD+Frontmatter, TOON, ISON, GOON, TRON, and more
-- **8 binary protocols** -- Protobuf, FlatBuffers, Cap'n Proto, MessagePack, CBOR, BSON, Avro, Thrift
-- **14 schema systems** -- JSON Schema, Zod, Effect Schema, Valibot, Pydantic, Standard Schema, and more
+## Schema Position
+
+Validators and schemas are treated as a separate layer.
+
+Current project stance:
+
+- validator libraries are not serialization formats
+- the strongest generation benefit usually comes from **schema strategy**, especially JSON Schema-oriented workflows
+- library choice mostly matters through:
+  - JSON Schema export quality
+  - runtime ergonomics
+  - ecosystem fit
+
+## Important Status Note
+
+Some benchmark content under `site/src/content/benchmarks/` and format metadata still exists primarily to support site UI development. Those placeholders should not be read as final reproduced benchmark results unless the corresponding run metadata is populated and `dummy` is false.
+
+## Repository Layout
+
+```text
+the-format-tax/
+├── README.md
+├── docs/
+│   ├── paper.md
+│   ├── decision-tree.md
+│   ├── benchmarks.md
+│   ├── schemas.md
+│   └── superpowers/
+├── benchmarks/
+│   ├── datasets/
+│   ├── questions/
+│   ├── references/
+│   └── runner/
+├── paper/
+└── site/
+```
 
 ## Development
 
-The landing page is an Astro site:
+### Site
 
 ```bash
 cd site
 bun install
 bun run dev
 ```
 
-## Contributing
-
-This is an open research project. We want:
-
-- New format benchmarks (especially on newer LLMs)
-- Schema system evaluations
-- Streaming parser comparisons
-- Real-world case studies ("we switched from X to Y and saved Z")
-- Corrections and challenges to our data
+### Benchmarks
 
-File an issue or open a PR. All benchmarks are reproducible.
+```bash
+cd benchmarks/runner
+bun run src/index.ts --track=1
+```
 
-## License
+The runner currently expects Bun.
 
-MIT (code) / CC BY-SA 4.0 (paper and documentation)
+## Contributing
 
-## Citation
+Useful contributions:
 
-```bibtex
-@article{formattax2026,
-  title={The Format Tax: How Serialization Choice Costs Tokens, Accuracy,
-         and Latency in Human--AI Systems},
-  author={Brunner, G{\"u}nther},
-  year={2026},
-  journal={arXiv preprint},
-  url={https://code-agents.github.io/format-tax}
-}
-```
+- independent reproductions of public format claims
+- better harness normalization
+- schema-guidance experiments
+- binary transport benchmark implementation
+- clearer separation between dummy site content and measured benchmark output
 
----
+## License
 
-*The Format Tax is a research project by [Günther Brunner](https://guntherbrunner.com) / [Code Agents](https://github.com/code-agents).*
-*Stop paying the tax. Start engineering your context.*
+MIT for code. Documentation and paper material follow the project’s existing documentation license conventions.

benchmarks/runner/package.json

@@ -7,7 +7,8 @@
     "bench:track1": "bun run src/index.ts --track=1",
     "bench:track2": "bun run src/index.ts --track=2",
     "bench:track3": "bun run src/index.ts --track=3",
-    "bench:track4": "bun run src/index.ts --track=4"
+    "bench:track4": "bun run src/index.ts --track=4",
+    "bench:track5": "bun run src/index.ts --track=5"
   },
   "dependencies": {
     "tiktoken": "^1.0.18",

benchmarks/runner/src/index.ts

@@ -1,9 +1,11 @@
 import { parseArgs } from "util";
-import { resolve, join } from "path";
+import { resolve } from "path";
 import { runTrack1 } from "./tracks/track1.ts";
 import { runTrack2 } from "./tracks/track2.ts";
 import { runTrack3 } from "./tracks/track3.ts";
 import { runTrack4 } from "./tracks/track4.ts";
+import { runTrack5 } from "./tracks/track5.ts";
+import { ensureOutputParent, resolveTrackOutputPath } from "./output-path.ts";
 
 const { values } = parseArgs({
   args: Bun.argv.slice(2),
@@ -12,7 +14,7 @@ const { values } = parseArgs({
     format: { type: "string", short: "f" },
     dataset: { type: "string", short: "d" },
     harness: { type: "string", short: "h" },
-    output: { type: "string", short: "o", default: "../../site/src/content/benchmarks" },
+    output: { type: "string", short: "o" },
     dry: { type: "boolean", default: false },
   },
   strict: true,
@@ -30,7 +32,7 @@ console.log(`  track:   ${values.track ?? "(all)"}`);
 console.log(`  format:  ${values.format ?? "(all)"}`);
 console.log(`  dataset: ${values.dataset ?? "(all)"}`);
 console.log(`  harness: ${values.harness ?? "(all)"}`);
-console.log(`  output:  ${values.output}`);
+console.log(`  output:  ${values.output ?? "../../site/src/content/benchmarks"}`);
 console.log(`  dry:     ${values.dry}`);
 console.log();
 
@@ -39,7 +41,7 @@ console.log();
 const RUNNER_DIR = import.meta.dir;
 const DATASET_DIR = resolve(RUNNER_DIR, "../../datasets");
 const QUESTION_DIR = resolve(RUNNER_DIR, "../../questions");
-const OUTPUT_DIR = resolve(RUNNER_DIR, values.output ?? "../../site/src/content/benchmarks");
+const DEFAULT_OUTPUT_DIR = resolve(RUNNER_DIR, "../../site/src/content/benchmarks");
 
 interface Question {
   id: string;
@@ -72,7 +74,7 @@ async function loadDatasets(filter?: string): Promise<Record<string, unknown>> {
     : [...DATASET_FILES];
 
   for (const name of names) {
-    const path = join(DATASET_DIR, `${name}.json`);
+    const path = resolve(DATASET_DIR, `${name}.json`);
     try {
       const file = Bun.file(path);
       datasets[name] = await file.json();
@@ -87,7 +89,7 @@ async function loadDatasets(filter?: string): Promise<Record<string, unknown>> {
 async function loadQuestions(): Promise<Question[]> {
   const allQuestions: Question[] = [];
   for (const name of QUESTION_FILES) {
-    const path = join(QUESTION_DIR, `${name}.json`);
+    const path = resolve(QUESTION_DIR, `${name}.json`);
     try {
       const file = Bun.file(path);
       const questions: Question[] = await file.json();
@@ -104,16 +106,16 @@ async function loadQuestions(): Promise<Question[]> {
 const tracksToRun: number[] = [];
 if (values.track) {
   if (values.track === "all") {
-    tracksToRun.push(1, 2, 3, 4);
+    tracksToRun.push(1, 2, 3, 4, 5);
   } else {
     for (const t of values.track.split(",")) {
       const n = parseInt(t.trim(), 10);
-      if (n >= 1 && n <= 4) tracksToRun.push(n);
+      if (n >= 1 && n <= 5) tracksToRun.push(n);
       else console.error(`  Unknown track: ${t}`);
     }
   }
 } else {
-  tracksToRun.push(1, 2, 3, 4);
+  tracksToRun.push(1, 2, 3, 4, 5);
 }
 
 async function main() {
@@ -125,7 +127,13 @@ async function main() {
   const dry = values.dry ?? false;
 
   for (const track of tracksToRun) {
-    const outputFile = join(OUTPUT_DIR, `track${track}.json`);
+    const outputFile = resolveTrackOutputPath({
+      baseDir: values.output ? process.cwd() : DEFAULT_OUTPUT_DIR,
+      outputArg: values.output ?? ".",
+      track,
+      totalTracks: tracksToRun.length,
+    });
+    await ensureOutputParent(outputFile);
 
     switch (track) {
       case 1:
@@ -165,6 +173,15 @@ async function main() {
         });
         break;
 
+      case 5:
+        await runTrack5({
+          harnessFilter: values.harness,
+          formatFilter: values.format,
+          outputPath: outputFile,
+          dry,
+        });
+        break;
+
       default:
         console.error(`Unknown track: ${track}`);
     }

benchmarks/runner/src/output-path.test.ts

@@ -0,0 +1,40 @@
+import test from "node:test";
+import assert from "node:assert/strict";
+
+import { resolveTrackOutputPath } from "./output-path.ts";
+
+const CLI_CWD = "/tmp/format-tax/benchmarks/runner";
+
+test("resolveTrackOutputPath keeps explicit json file path for single-track runs", () => {
+  const outputPath = resolveTrackOutputPath({
+    baseDir: CLI_CWD,
+    outputArg: "../tmp-bench/custom-track3.json",
+    track: 3,
+    totalTracks: 1,
+  });
+
+  assert.equal(outputPath, "/tmp/format-tax/benchmarks/tmp-bench/custom-track3.json");
+});
+
+test("resolveTrackOutputPath appends track filename when output is a directory", () => {
+  const outputPath = resolveTrackOutputPath({
+    baseDir: CLI_CWD,
+    outputArg: "../tmp-bench",
+    track: 3,
+    totalTracks: 2,
+  });
+
+  assert.equal(outputPath, "/tmp/format-tax/benchmarks/tmp-bench/track3.json");
+});
+
+test("resolveTrackOutputPath rejects json file outputs for multi-track runs", () => {
+  assert.throws(
+    () => resolveTrackOutputPath({
+      baseDir: CLI_CWD,
+      outputArg: "../tmp-bench/custom.json",
+      track: 1,
+      totalTracks: 2,
+    }),
+    /select exactly one track/i,
+  );
+});