Commit bfe88f1
committed
feat(mcp): wire MCP tools to real tok package and align docs to library-only
Three changes that together bring tok's MCP surface and its docs
in line with the actual library-only codebase.
mcp/server.go — the existing implementation was a stub that
hard-coded:
- count_tokens: len(text)/4 (no BPE)
- estimate_cost: return 0.0 (no pricing)
- compress_text: collapseWS (no pipeline)
Replace each handler with a real call into the public tok API:
- count_tokens(text, model) → tok.EstimateTokens / EstimateTokensForModel
- estimate_cost(model, in, out) → tok.GetModelPricing × (in/1000*ip + out/1000*op)
- compress_text(text, mode, budget?) → tok.Compress with preset Option
- redact_secrets(text, entropy?) → tok.NewSecretDetector (new tool)
Add a 4th tool redact_secrets backed by the 33-pattern secret
detector. The legacy collapseWS helper is removed.
mcp/server_test.go — replace the stub tests with real-pipeline
tests. The 4x token reduction assertion on aggressive mode is the
stub-regression guard: a future refactor that re-introduces
len/4 / 0.0 / collapseWS shortcuts fails CI.
ARCHITECTURE.md — still claimed 'Go library and CLI tool' and
drew a Tok CLI box with command runner / filter selector / rewrite
engine. Also referenced agents/ and hooks/ directories that don't
exist, and a 'Command Rewriting' flow that was never library-side.
Rewrite to library-only architecture, drop the CLI box and the
Command Rewriting section, update the file structure table to
match reality, update Security to 33 patterns, update Build &
Release to library distribution via 'go get'.
CITATION.cff — title and abstract claimed 'Token-Aware CLI Proxy'.
Update to 'A Go Library for Prompt Compression, Output Filtering,
Token Estimation, and Secret Detection' and reword the abstract
to describe the library API, the 31-layer pipeline, and the
go-get distribution path.1 parent e61c143 commit bfe88f1
4 files changed
Lines changed: 723 additions & 182 deletions
0 commit comments