Empirical evaluation: 30-run benchmark of token-efficient vs 4 other .claude/ configs

[adam-s]

Summary

I built a deterministic evaluation harness to test whether aggressive output-reduction rules actually save total tokens in agentic coding tasks. Your repo's actual CLAUDE.md was tested directly alongside 5 other configurations across 3 coding challenges.

Each agent gets a test file and must make all tests pass. All configs pass 100%. The comparison is purely tokens to green.

The 6 Configs Tested

Config	What's in .claude/	Size
A-baseline	"A coding project."	1 line
B-token-efficient	Our 12-line summary of token-reduction ideas	12 lines
C-structured	CLAUDE.md + rules + agents + reference	4 files
D-workflow	CLAUDE.md + rules + skills + hooks	4 files
E-hybrid	CLAUDE.md + rules + agents	3 files
F-drona23	Your actual CLAUDE.md from this repo	61 lines

Results — All Pass, Token Cost Varies

CSV Reporter

Config	Avg Tokens	Avg Cost
E-hybrid	1,012	$0.068
C-structured	1,016	$0.067
A-baseline	1,088	$0.078
B-token-efficient	1,096	$0.093
F-drona23	1,137	$0.084
D-workflow	1,199	$0.083

SQLite Window Functions

Config	Avg Tokens	Avg Cost
E-hybrid	1,230	$0.108
A-baseline	1,255	$0.120
C-structured	1,287	$0.116
B-token-efficient	1,339	$0.116
D-workflow	1,374	$0.123
F-drona23	1,586	$0.127

Hono WebSocket Counter (hardest — multi-turn debugging)

Config	Rep 1	Rep 2	Avg Tokens	Avg Cost
B-token-efficient	3,370	—	3,370	$0.136
A-baseline	3,667	—	3,667	$0.160
E-hybrid	4,121	—	4,121	$0.151
C-structured	4,778	—	4,778	$0.187
D-workflow	5,334	—	5,334	$0.182
F-drona23	14,182	4,478	9,330	$0.268

Honest Assessment

Your approach works — the ideas are sound. Eliminating sycophancy, reducing redundant output, and keeping code simple are all good principles. Our 12-line summary of these ideas (Config B) performed well, especially on the hardest task (WebSocket: 3,370 tokens, cheapest of all).

But the actual 61-line CLAUDE.md from this repo doesn't work as well as a shorter version of the same ideas. The file adds significant input token overhead on every turn. On simple tasks (CSV, SQLite), it's 10-20% more expensive than baseline. On the WebSocket challenge, it hit 14,182 tokens in one run — 4x more than any other config.

The 61 lines include detailed rules about typography, speculation control, session memory, scope control, and warnings. These are all reasonable rules, but each one costs input tokens on every single API turn. The model already follows most of these behaviors by default — explicitly stating them adds cost without changing output.

The sweet spot is fewer, higher-impact rules. Our best-performing config (E-hybrid, $0.327 total) uses 7 lines:

- Think before acting. Read existing files before writing code.
- Be concise in output but thorough in reasoning.
- Prefer editing over rewriting whole files.
- Do not re-read files you have already read.
- Test your code before declaring done.
- No sycophantic openers or closing fluff.
- Keep solutions simple and direct.

Methodology

• Isolated git worktrees — agents see only seed files + test file + their .claude/ config
• Same model (claude-sonnet-4-6) for all runs
• All produced code saved for inspection
• Repo: https://github.com/adam-s/testing-claude-agent

[drona23]

Thanks for building this. Genuinely useful harness.

We iterated through v2-v8 using your benchmark. The original 61-line file (F-drona23) was bloated, you were right about that. We trimmed it down to 7 lines across 2 files.

Ran v8 head-to-head against C-structured on the same harness, same day, same model:

Challenge	M-drona23-v8	C-structured
CSV Reporter	$0.244	$0.282
SQLite Windows	$0.406	$0.376
WebSocket	$0.285	$0.473
Total	$0.935	$1.131

v8 wins overall by 17.4%. The WebSocket gap is the biggest because v8 has an explicit rule to use a Set for client tracking instead of pub/sub, which prevents the debugging loop that costs extra turns.

Key takeaway we learned from your data: cost scales with turns, not file size. Each turn adds ~40k cache_read tokens. The one-shot strategy (read everything, write complete solution, test once) matters more than what's in the CLAUDE.md.

Root CLAUDE.md is now 8 lines. Benchmark configs are in profiles/.

Thanks again for the harness. Updated our README with the head-to-head results and linked your repo.

[drona23]

Quick update tied to this thread: just merged #10 (reproducible benchmark harness) and re-ran on the current minimal CLAUDE.md.

benchmark/run.py now drives claude -p --output-format json in isolated temp dirs across haiku/sonnet/opus at N=5, capturing real output_tokens. On the current minimal profile the measured output-token reduction is roughly haiku -2%, sonnet -11%, opus -7% (excluding T4). With profiles/CLAUDE.compressed.md (also shipped today), opus hits -62%.

This doesn't replicate your 30-run cost-to-green coding harness, so leaving the issue open. The harness here is the single-shot Q&A side; yours covers the agentic side. Linking them as complementary.

• Fresh per-model reports: benchmark/report-{haiku,sonnet,opus}.md
• Token summary: benchmark/SUMMARY.md
• Raw per-run records: benchmark/raw-*.jsonl

[adam-s]

@drona23 Along the same path, I did some exploration into understanding how thinking models solve problems. More or less, I asked the same model the same question and analyzed the thinking output over several iterations. You might be interested.

https://adamsohn.com/reliably-incorrect/ and https://adamsohn.com/lambda-variance/

[drona23]

Read both. The p_step^N framework lines up with what our N=5 semantic eval showed.

In SEMANTIC.md from #10, judge accuracy and completeness are 100% in both conditions across haiku, sonnet, opus. Em-dash and verbosity markers move; correctness does not. The rules change tone, not capability. That matches your point: p_step is set by what's dense in training, not by the system prompt.

The verification-on-same-weights observation is the cleanest framing I've seen for why "check your work" rules in CLAUDE.md add tokens without adding signal. The fabrication guard from #9 runs into exactly this wall.

Lambda-variance is the harder result for benchmark methodology. N=5 catches output-length variance, but if the underlying strategy varies run-to-run, mean output_tokens is a thinner signal than I'd been treating it as. Going to look at our raw-*.jsonl traces with that lens.

Thanks for sharing. Bookmarked the agent-tuning repo too.