feat(compression): full overhaul — pipeline, metrics, idempotency, robustness

[manthis]

Summary

Reworks the compression layer end-to-end. Two main goals:

1. Stop silently breaking the upstream prompt cache. The previous compressor
   would re-process every tool message on every turn — any change in
   compression behaviour (bug fix, threshold tweak) would invalidate the
   Anthropic / OpenAI prompt cache for the whole conversation.
2. Open the architecture for new techniques. The crate description has
   always promised "multiple composable techniques", but only one was wired
   in and the dispatch was hard-coded. This PR introduces the trait + chain
   and demonstrates it with a second technique.

Along the way, fixes a handful of correctness gaps (<tool_use_error> false
positives, lost Codex exit codes), tightens dispatch on real-world bash
invocations (sudo cargo build, cd src && cargo test, /usr/local/bin/cargo),
adds per-tool metrics, image-resistant memory bounds, and a benchmark suite.

14 commits, +2 100 / −80 lines, 49 new tests (386 → 435), zero clippy
warnings under -D warnings.

Real-world results

End-to-end integration test (tests/integration.rs) on a fixture mirroring a
mid-session Claude Code request — 3.5 KB system prompt + 4-turn dialogue with
Read / Glob / Bash / Grep tool calls totalling ~75 KB of tool output:

system prompt: 3 602 bytes  → cache_control: ephemeral injected
tool inputs:   75 394 bytes
tool outputs:  30 652 bytes
saved:         44 742 bytes (59 %)

Per-tool breakdown:

Tool	Input	Output	Saved
Bash (cargo build)	3 892	220	94 %
Glob (250 paths)	5 139	619	88 %
Grep (240 matches)	13 380	1 859	86 %
Read (800-line Rust)	52 983	27 954	47 %

Idempotency verified: a second pass through the pipeline is a byte-for-byte
no-op, so re-encoding never wakes the prompt cache.

What changed

Core correctness & cache safety

• Versioned output marker — every compressed result is prefixed with
  <!--ec1-->. Re-entry detects the marker and short-circuits, so a stable
  compressed message stays byte-identical across turns and the upstream
  prompt cache survives.
• Memory guard — MAX_COMPRESSIBLE_BYTES = 2 MiB. Pathological tool
  outputs no longer turn a single request into a DoS vector.
• Stricter protected-tag detection — <tool_use_error> /
  <persisted-output> are now anchored to line start. Documents and code
  that mention the tags compress normally instead of being silently
  rejected.
• Codex exit code preserved — non-zero exits from shell_command are
  re-injected as [exit N] after the marker, so the agent still sees
  failure signal even after the header is stripped.

Architecture

• CompressionTechnique trait + CompressionPipeline — composable chain
  applied in order to every CompletionRequest. Existing tool-result
  compression becomes ToolResultsTechnique. CompressionLayer::new keeps
  the previous behaviour drop-in; CompressionLayer::with_pipeline is the
  new escape hatch for custom chains.
• SystemPromptCacheTechnique (new) — auto-injects
  cache_control: {"type": "ephemeral"} on large, un-hinted system messages
  so Anthropic can serve them from prompt cache. Caps total injections to
  stay under the 4-breakpoint limit, counts pre-existing hints to remain
  idempotent across passes.

Bash dispatch

Real-world tool calls now route correctly:

• Strip leading env-var assignments (FOO=bar cargo build).
• Strip wrapper keywords: sudo, time, env, nohup, exec.
• Peel silent leading sub-commands followed by && / ; / ||
  (cd src && cargo build, export X=1 && cargo test).
• Dispatch by basename so /usr/local/bin/cargo and bare cargo route
  identically.

Read enhancements

• New language coverage: TOML, JSON, YAML, Markdown. JSON / Markdown skip
  comment-stripping entirely so values that look like comments survive.
• Lockfile detection (Cargo.lock, package-lock.json, yarn.lock,
  pnpm-lock.yaml, go.sum, …) — replaced with a head + tail + elision
  stub. Generated lockfiles eat token budget for almost no informational
  value.
• Aggressive mode for brace-language files >500 lines: collapse function /
  class bodies of 8+ lines into a // ... (N lines collapsed) placeholder.
  A 5 000-line source file becomes a usable skeleton instead of a slightly
  smaller wall of text.
• Threshold relaxed: accept either ≥10 % savings or ≥200 bytes saved
  (whichever fires first). Stops rejecting modest absolute gains on large
  files.

Observability

• CompressionMetrics — per-tool counters (invocations, skipped,
  bytes_in, bytes_out) shared across cloned layer / service handles
  via Arc. Snapshot + totals APIs return owned data, ready for export
  via a /metrics HTTP handler.
• edgee stats --per-tool — aggregates tool_compression_stats from
  every stored session log, sorted by absolute savings descending. Useful
  both for tuning and for spotting tools where the compressor does
  nothing.

Performance & test infrastructure

• Regex early-out in split_into_segments — skip the regex walk
  entirely when the literal <system-reminder> substring is absent
  (>99 % of inputs).
• Criterion benchmarks — six micro-benches covering Bash / Read / Grep /
  Glob / segment-protection / Codex pipeline. cargo bench -p edgee-compressor.

Estimated cost per improvement

Cost dimensions per change:

• Latency — added time per request on the hot path.
• Memory — bytes allocated per request beyond what we had before.
• Tokens — bytes added to (or removed from) the wire payload sent to the
  provider, multiplied by realistic conversation length.
• Code — net SLOC added and conceptual surface to maintain.

Latency numbers below are taken from cargo bench -p edgee-compressor -- --quick on an Apple Silicon dev machine, measured per tool message.

Improvement	Latency	Memory	Tokens	Code
Memory guard (MAX_COMPRESSIBLE_BYTES)	~1 ns (len() compare)	0	0	+30 SLOC
Versioned marker (<!--ec1-->)	~50 ns (one starts_with + one format!)	+10 bytes per compressed message (one String grow)	+10 bytes ≈ 3 tokens per tool message; pays for itself in 1 cache hit on the next turn	+50 SLOC
Stricter tag detection	+5–50 µs on outputs that don't carry the tags (lines().any instead of contains)	0	0	+10 SLOC
Codex exit-code preservation	~80 ns header scan; one format! only on non-zero exits	+10 bytes when exit ≠ 0	+~5 tokens on failed shell calls only	+30 SLOC
Bash dispatch (prefixes / basename / silent chains)	~150 ns of byte scanning per Bash tool call	0 (all &str slices)	0	+160 SLOC
Read: new langs + threshold relax	~0 (added match arms)	0	Negative — unlocks compression that was previously rejected	+15 SLOC
Read: lockfile stubs	O(1) basename match; format!() only on hit	One small String on hit	Large negative — typical Cargo.lock shrinks from 5–50 KB to ~200 bytes	+60 SLOC
Read: aggressive brace collapse	O(n) brace scan, only when file > 500 lines AND brace-language; ~30 µs on a 1 000-line file	One Vec reallocated to filtered size	Large negative — a 5 000-line file collapses to a few hundred lines of skeleton	+120 SLOC
Regex early-out	Negative latency — skips a 1–10 µs regex walk on the >99 % of outputs without a <system-reminder>	0	0	+5 SLOC
CompressionTechnique pipeline	~5 ns per technique (Vec iter + dyn dispatch)	One Arc<CompressionPipeline> per layer (shared)	0	+130 SLOC
SystemPromptCacheTechnique	~1 µs per request (one walk over messages)	One serde_json::Value per injection (≤2 per request)	0 added; enables 90 %+ token-cost discount on cached system blocks via Anthropic prompt cache	+220 SLOC
Per-tool CompressionMetrics	~200–500 ns per tool message (one mutex acquire + one HashMap::entry)	One String (tool name) on first encounter, then nothing	0	+180 SLOC
edgee stats --per-tool	Out of hot path — only runs on CLI invocation	Negligible	0	+100 SLOC
Criterion benches	Out of hot path — dev-only	0	0	+310 SLOC (dev-only)

Concrete bench results on the fixture in tests/integration.rs:

bash_git_diff_40_hunks            22 µs
read_rust_800_lines              114 µs
grep_content_2000_matches        234 ms   (pre-existing algorithm; flagged for follow-up)
glob_400_paths                    55 µs
segment_protection_no_reminder    28 µs   (the early-out path)
codex_shell_command_200_files     52 µs

Net per-request overhead is well under a millisecond for everything except
the Grep content strategy, which was already this slow before this PR
(2 000 matches × O(matches) regrouping). Filing a follow-up to revisit it.

Test plan

• cargo fmt --all
• cargo clippy --all-targets -- -D warnings — zero warnings
• cargo test --all — 435 passed, 0 failed
• cargo bench -p edgee-compressor --no-run — benches compile
• cargo bench -p edgee-compressor -- --quick — benches execute, numbers above
• cargo test -p edgee-compression-layer --test integration -- --nocapture
  — 59 % byte savings on a realistic fixture, idempotency confirmed
• Smoke test against a live Claude Code session — verify the marker
  survives round-trips and the Anthropic cache_creation / cache_read
  counters reflect the SystemPromptCacheTechnique injections
• Confirm edgee stats --per-tool against an existing session log
  directory

Backwards compatibility

• CompressionLayer::new(config) still exists and produces a default
  pipeline of [tool-results] — no behavioural change for current callers.
• CompressionConfig::new(agent) returns the same Arc<CompressionConfig>
  as before; the new metrics field defaults to a fresh
  Arc<CompressionMetrics>.
• All public re-exports from previous releases are preserved
  (compress_tool_output, compress_codex_tool_output, claude_compressor_for,
  …).

Future work (deliberately out of scope)

• Image down-sampling (needs image crate dependency, invasive content-type
  handling).
• Tool-call arguments dedup via content-hash references (request-level
  state).
• Async LLM-based conversation roll-up for old turns.
• Fuzzing the parsers via cargo-fuzz / proptest.
• Per-conversation LRU cache to avoid re-indexing on every request.
• Revisit grep content mode performance — 234 ms on 2 000 matches is the
  one outlier in the bench suite (pre-existing).

---

From HelloMax to Edgee with 🫶

[KokaKiwi]

Thanks for the PR, lots of nice stuff in here. The per-tool metrics, the memory guard, and SystemPromptCacheTechnique all fill real gaps, and the jump from 386 to 435 tests with zero clippy warnings is appreciated.

Flagging the review with "Request changes" because of the brace-collapse compression on crates/compressor/src/strategy/claude/read.rs that could silently drop code when braces appear inside string literals.
We previously had some "aggressive" Read compression as well but it seemed to incapacitate the coding agent a lot when doing that so we had to tone down a bit this tool compression specifically.

Rest is smaller reviews plus one design question I would love your take on.

[KokaKiwi]

The brace counter on lines 547-560 walks every { / } character without filtering string literals, raw strings, or block comments. If a body contains a literal brace, depth reaches zero inside the literal, closed is set, and everything between that false close and the real } is silently dropped.

Repro shape:

fn build_query() {
    let sql = r#"SELECT * FROM t WHERE x = '}'"#;
    actual_code_here();
}

The doc-comment above the function says mis-counts only cause "kept too much, never silent data loss". That contract is violated here.

Tightening the boundary check is enough to make this sound. We accept fewer collapses (when } shares a line) but never drop code:

let closing = lines[j].1.trim();
if closed && j > i && matches!(closing, "}" | "};" | "},") {
    // safe to collapse
}

While we're here, the docstring at lines 530-531 needs to be updated too.

---

Review comment authored with Claude Code (Opus 4.7).

[KokaKiwi]

All four public methods of CompressionMetrics (lines 54, 68, 80, 92) lock the mutex with .expect("compression metrics mutex poisoned"). Once any thread panics while holding the lock, every subsequent request on the hot path panics as well, a single observability-path issue takes the whole gateway down.

For accumulating counters, a partial / stale read is always preferable to a crash. The standard recovery is enough here:

self.inner.lock().unwrap_or_else(|e| e.into_inner())

Apply to all four call sites.

---

Review comment authored with Claude Code (Opus 4.7).

[KokaKiwi]

The doc says "system / developer message" but the implementation only matches Message::System (line 93), and DeveloperMessage doesn't have a cache_control field anyway. Either drop "developer" from the doc, or add the field in gateway-core and a Message::Developer arm in the loop and the counter.

[KokaKiwi]

The CompressionTechnique trait + CompressionPipeline design works and makes sense for the near-term roadmap (MCP cleaning, caveman summarization, etc.). The builder API (CompressionLayer::with_pipeline) is a nice escape hatch too.

One thing I had in mind when thinking about this was using Tower layers directly, each technique as its own tower::Layer in the ServiceBuilder chain, which would keep the composition model consistent with the rest of the gateway stack. That said, I think there's a middle ground: we could keep the internal CompressionPipeline as-is and bridge it to Tower properly in a follow-up, once we have a clearer picture of what the next techniques look like. Not asking for changes here, just flagging it as something worth revisiting post-merge.

[KokaKiwi]

I want to double-check the assumption behind is_already_compressed and the COMPRESSION_MARKER prefix.

The stated goal is cache-safety: if the same compressed message re-enters the pipeline unchanged, the upstream Anthropic prompt cache sees a stable byte sequence. That logic holds if the gateway ever receives its own compressed output back, but in our proxy model, it doesn't. The gateway compresses tool results before forwarding to Anthropic, but the compressed form is never echoed back to the client. On every subsequent turn the client resends its own copy of the conversation history with the original, uncompressed tool outputs, so the gateway always starts fresh.

The scenario the marker is protecting against (a previously compressed message re-entering the pipeline) can't happen structurally, so the guard feels un-needed. And even if we wanted that protection at the compressor crate level, it seems more natural for the crate's caller to be responsible for not feeding already-compressed content back in, rather than baking the marker into the output format.

Happy to be convinced otherwise if there's a flow I'm missing.