Reversible context compression (ctxzip): 60-95% fewer tokens on bulky tool outputs, losslessly

[initializ-mk]

What

Integrates ctxzip v0.1.0 — reversible, structure-aware context compression — into the agent loop. Bulky tool outputs (JSON arrays, logs, grep results) are compressed before reaching the LLM; everything dropped is stored in a durable local bbolt store behind an inline <<ctxzip:HASH ...>> marker and retrievable via the new context_expand tool. Lossy on the wire, lossless end-to-end.

Off by default. Enable via compression.enabled: true in forge.yaml, FORGE_COMPRESSION=true, forge run --compression, or the new init-wizard step.

Architecture — three seams (forge-core/compress/)

1. AfterToolExecHook — compresses tool output once, at production time, before it enters Memory. Compressed bytes never change afterwards, so the conversation prefix stays byte-stable and provider prompt caches keep hitting. Registered after guardrail/redaction hooks; error results and small outputs stay verbatim.
2. WrapClient — llm.Client decorator below the FallbackChain (covers retries + compactor calls) compressing the live zone of each request. Deterministic across turns: relevance query pinned to the first user message, never the latest turn.
3. ExpandTool — context_expand(hash) builtin; the loop executes it like any other tool, no retrieval machinery needed. Tolerates imperfect hashes (whole markers, truncated hex → unique-prefix resolution against recently emitted markers).

A runtime-owned system directive is appended when compression is on, so every skill's agent knows what markers are and when to expand — skill authors need zero awareness.

Provider prompt-cache hints (ClientConfig.PromptCaching)

Gated by compression.cache_hints (defaults to enabled):

• anthropic: cache_control: ephemeral breakpoints on the last tool definition + system block (block-form system only when on — wire format byte-identical to today when off, test-asserted). Also applies on the aws_sigv4 Bedrock-passthrough path.
• openai/gemini: stable prompt_cache_key derived from (model, system, tool names).

Observability

• context_compressed / context_expanded audit events (via EmitFromContext — correlation_id/task_id/seq/signing like every other event) carrying per-event figures plus running totals.
• invocation_complete gains compression_saved_tokens_total, compression_count, expansion_count — per-invocation, keyed by correlation ID so concurrent tasks never cross-contaminate.

Config & CLI surfaces

compression:
  enabled: true
  keep_patterns: [CrashLoopBackOff, PAYMENT_DECLINED]  # domain never-drop vocabulary
  # store_path (.forge/ctxzip.db) / ttl (30m) / min_tool_output_chars (2048) / cache_hints

• forge run --compression[=false] — tri-state override (absent = yaml/env decide)
• forge serve --compression[=false] — forwarded to the daemon
• forge init --compression + a new Context Compression TUI wizard step

Live-tested

Hardened against a real gpt-4o agent over a 150-pod fixture (several commits exist because live testing found the failure modes — expand/compress tail-chase, hash transcription, store-dir creation):

• grep output compressed 1,397 → 51 tokens (96%) with the one CrashLoopBackOff pod kept verbatim
• model called context_expand unprompted, hash transcribed correctly, got intact lines back, answered with the exact error (OOMKilled 512Mi)
• surgical sessions correctly report compression_count: 0 — compression is insurance against bulk, not a tax on every call

Notes for reviewers

• Fail-open everywhere: store init failure, compression error, or inflation all fall back to uncompressed originals.
• Token figures in audit fields are tokenizer estimates (documented); billed truth stays in llm_call.input_tokens.
• Known follow-ups (intentionally out of scope): compression-aware tool limits (grep's internal 50-line default truncates upstream of the reversible layer); grep_search returning "(no matches found)" for a nonexistent file; tools: in forge.yaml not registering builtins (banner is cosmetic).

🤖 Generated with Claude Code

[initializ-mk]

Requesting changes (posting as a comment — GitHub won't let the author submit a formal Request-Changes review on their own PR). Blocking item: the streaming perInvocation leak below.

Reviewed for correctness and traced the invocation-complete paths + concurrency. Strong PR — fail-open everywhere, careful locking (accumulate under lock → snapshot → emit audit outside lock), the expand-tool-output-never-recompressed tail-chase fix, and the "wire format byte-identical when off" prompt-cache path is test-asserted. Builds clean; compress + llm/providers tests pass. One real bug on the primary path.

🔴 perInvocation leaks (and compression metrics are missing) on the streaming paths

TakeInvocationTotals — which both pops perInvocation[correlationID] and adds the compression_* fields — is called in exactly one place: executeTask (runner.go:1552). But there are three invocation_complete emission sites, and the other two are the sendSubscribe streaming handlers that run their own ExecuteStream loop instead of executeTask:

• tasks/sendSubscribe (JSON-RPC SSE) → emits at 1296, no Take.
• REST POST /tasks/sendSubscribe → emits at 1821, no Take.

Both streaming handlers set a correlation ID on ctx (WithCorrelationID) and use the shared compressed client + hooks, so a streaming invocation that compresses does call recordCompression → bumpInvocation, adding a perInvocation[cid] entry. Since only executeTask pops, and perInvocation has no TTL/sweep, every streaming invocation that compresses leaks its bucket permanently. sendSubscribe is a primary A2A mode, so with compression on this grows unbounded over process lifetime.

Same root cause, second gap: the PR advertises invocation_complete gaining compression_saved_tokens_total / compression_count / expansion_count, but those are only added in executeTask — so streaming invocation_complete events don't carry them. The feature is under-delivered on the common path.

Fix (one change closes both): factor the "TakeInvocationTotals(ctx) + populate the three compression fields" block out of executeTask into a helper, and call it at all three EmitInvocationComplete sites (1296, 1565, 1821). Optionally also give perInvocation a size cap / periodic sweep so a missed Take can't leak — but the shared helper is the real fix and also ships the metrics.

🟡 Secondary

• recent marker map is unbounded — every emitted marker hash is remembered (rememberMarkers) and never evicted. Small strings, but grows for the process lifetime; a cap / LRU would bound it (you only need recent markers for imperfect-hash resolution).
• bbolt is a single-writer file lock. .forge/ctxzip.db can be opened by only one process; two agents on a shared volume (K8s multi-replica) or forge run + forge serve on the same dir will collide. Fail-open only helps if ccr.NewBoltStore errors rather than blocks on the flock — worth confirming it sets an Open timeout, and documenting the single-writer constraint in the compression doc ("one store per process/replica").
• Inflation guard is == 0, not <= 0 (hook.go: res.SavedTokens() == 0). If ctxzip can ever return negative savings (inflated output), that guard misses it and you'd record negative savings + apply the inflated bytes. Depends on ctxzip's contract; low risk, but <= 0 is safer.

Note (in the PR's favor)

go.mod is correct — ctxzip is a direct require and bbolt is correctly // indirect (transitive via ctxzip; compress imports ctxzip/ccr, not bbolt). No go mod tidy needed.

Verdict

Request changes for the streaming perInvocation leak + missing metrics — a genuine unbounded leak on the main path and a gap between advertised and actual streaming behavior. The shared-helper refactor is small and fixes both. The bbolt single-writer note and recent cap are worth doing but non-blocking.

[initializ-mk]

Resolved in deb06fd — all four findings addressed:

🔴 Streaming perInvocation leak + missing metrics — fixed with the suggested shared helper. The pop+populate block is now appendCompressionFields(ctx, fields), called at all three EmitInvocationComplete sites (executeTask, JSON-RPC SSE sendSubscribe, REST sendSubscribe). The helper's doc comment names all three sites explicitly so a future fourth emission path knows the contract. Pinned by TestAppendCompressionFields_PopsAndPopulates: fields populated on first call, one-shot pop (second call returns zeros — no double-count if two sites ever fire), nil-safe with compression disabled.

🟡 Leak backstops (took the "optionally also" suggestion): perInvocation is now bounded to 1024 buckets with oldest-touched eviction — a future missed pop degrades to bounded memory instead of an unbounded leak. recent markers bounded to 2048 with oldest-emitted eviction (only recent markers matter for transcription repair; exact hashes still resolve via the store). Both pinned by tests (TestPerInvocationBuckets_Bounded, TestRecentMarkers_Bounded).

🟡 Inflation guards: both seams tightened from == 0 to <= 0. Confirmed ctxzip's SavedTokens() clamps at zero today, so this is defense-in-depth against a future contract change — noted as such in the code comment.

🟡 bbolt single-writer: confirmed ccr.NewBoltStore sets bolt.Options{Timeout: 5 * time.Second} — a second process errors after 5s rather than blocking, so fail-open engages (warning logged, agent runs uncompressed). Documented in context-compression.md § Failure posture: one store per process/replica, give each replica its own store_path.

Gate: build ✅, vet ✅, gofmt ✅, forge-core 32 packages ✅, forge-cli/runtime full suite ✅.

And thanks for the go.mod note — glad the direct/indirect split checked out.

🤖 Generated with Claude Code