Skip to content

Commit bfe88f1

Browse files
committed
feat(mcp): wire MCP tools to real tok package and align docs to library-only
Three changes that together bring tok's MCP surface and its docs in line with the actual library-only codebase. mcp/server.go — the existing implementation was a stub that hard-coded: - count_tokens: len(text)/4 (no BPE) - estimate_cost: return 0.0 (no pricing) - compress_text: collapseWS (no pipeline) Replace each handler with a real call into the public tok API: - count_tokens(text, model) → tok.EstimateTokens / EstimateTokensForModel - estimate_cost(model, in, out) → tok.GetModelPricing × (in/1000*ip + out/1000*op) - compress_text(text, mode, budget?) → tok.Compress with preset Option - redact_secrets(text, entropy?) → tok.NewSecretDetector (new tool) Add a 4th tool redact_secrets backed by the 33-pattern secret detector. The legacy collapseWS helper is removed. mcp/server_test.go — replace the stub tests with real-pipeline tests. The 4x token reduction assertion on aggressive mode is the stub-regression guard: a future refactor that re-introduces len/4 / 0.0 / collapseWS shortcuts fails CI. ARCHITECTURE.md — still claimed 'Go library and CLI tool' and drew a Tok CLI box with command runner / filter selector / rewrite engine. Also referenced agents/ and hooks/ directories that don't exist, and a 'Command Rewriting' flow that was never library-side. Rewrite to library-only architecture, drop the CLI box and the Command Rewriting section, update the file structure table to match reality, update Security to 33 patterns, update Build & Release to library distribution via 'go get'. CITATION.cff — title and abstract claimed 'Token-Aware CLI Proxy'. Update to 'A Go Library for Prompt Compression, Output Filtering, Token Estimation, and Secret Detection' and reword the abstract to describe the library API, the 31-layer pipeline, and the go-get distribution path.
1 parent e61c143 commit bfe88f1

4 files changed

Lines changed: 723 additions & 182 deletions

File tree

0 commit comments

Comments
 (0)