docs(tok): update README and CHANGELOG with Round 2 porting additions

Document the 8 new features added in the Round 2 rtk + caveman porting
session (2026-06-01):

  - tok.CompressCaveman (Lite/Full/Ultra)
  - tok.IsSensitiveFilename (3-layer detection)
  - filter.SmartTruncate (rtk kept+dropped==total invariant)
  - tok.ExtractJSON/ExtractJSONArray/ExtractAllJSON
  - tok.NewTracker (SQLite gain tracker)
  - tok.EstimateTokensFast/WithEncoding/ForModel
  - filter.CompressWithRetry (validate-fix-retry loop)
  - filter.NewTOMLFilter (8-stage rtk pipeline)
  - RetryStats.FinalPipelineStats pointer fix (vet copylocks)

The README's "Recent additions" section is split into two
chronological sections (2026-06-01 and 2026-04-20) so the latest
work is on top. The CHANGELOG adds a new "Round 2 of rtk + caveman
porting" subsection under [Unreleased].

Author: Patel230

CHANGELOG.md

@@ -16,6 +16,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Dollar-based cost savings tracking in Stats
 - Model-aware BPE encoding selection
 
+### Added — Round 2 of rtk + caveman porting (2026-06-01)
+- **`tok.CompressCaveman(text, intensity)`** — public Go API for the
+  caveman prompt-compression algorithm. Three intensity levels
+  (`CavemanLite`, `CavemanFull`, `CavemanUltra`) with drop-lists for
+  articles / filler / pleasantries and ~150 phrase substitutions.
+  Auto-clarity: security / destructive segments pass through verbatim.
+  Returns a `CavemanStats` struct (OriginalBytes, CompressedBytes,
+  BytesSaved, PercentOff, PassThroughSegments, etc.).
+- **`tok.IsSensitiveFilename(path)`** — 3-layer path-based sensitive
+  detection (exact basename, sensitive directory, name token).
+  Companion to the content-based `SecretDetector`.
+  Categories: `CatExactBasename`, `CatSensitiveDirectory`, `CatNameToken`.
+- **`filter.SmartTruncate`** with the rtk `kept + dropped == total`
+  invariant. Returns `TruncateStats` so callers can verify the
+  accounting from the output text alone.
+- **`tok.ExtractJSON` / `tok.ExtractJSONArray` / `tok.ExtractAllJSON`** —
+  brace-balanced JSON extractor. Handles LLM output with surrounding
+  prose, markdown code fences, and unterminated objects.
+- **`tok.NewTracker(ctx)`** — persistent SQLite gain tracker
+  (WAL mode, 90-day retention, pure Go via `modernc.org/sqlite`).
+  Records per-event compression stats and supports aggregate
+  queries. Default path `~/.tok/tracker.db`.
+- **`tok.EstimateTokensFast/WithEncoding/ForModel`** — model-aware
+  token estimation exposed at the top level (was previously internal).
+- **`filter.CompressWithRetry`** — validate-fix-retry loop. Caller
+  supplies a `Validator` and `AdjustFunc`; the loop escalates
+  mode/intensity and retries up to `MaxRetries` times. Includes
+  `MustContainsValidator` and `MaxTokensValidator` helpers.
+- **`filter.NewTOMLFilter` / `LoadTOMLFilterFile`** — full rtk
+  8-stage pipeline (strip_ansi, replace regex, match_output
+  short-circuit, strip/keep_lines, truncate_lines, head/tail,
+  max_lines, on_empty) as a `Filter` implementation. Plugs into
+  the main pipeline via the existing `tomlFilterWrapper` slot.
+- Bug fix: `RetryStats.FinalPipelineStats` is now `*PipelineStats`
+  to avoid copying `sync.RWMutex` per Go vet `copylocks` warning.
+
 ### Added — Production Hardening (top-50 OSS parity)
 - Same-style hardening pass already on this branch:
   strict `golangci-lint` v2 config (`errcheck`, `staticcheck`, `gocritic`

README.md

@@ -130,6 +130,19 @@ Reproduce: `go build -o tok ./cmd/tok && TOK=./tok evals/bench.sh --no-rtk`
 
 ## Recent additions
 
+Session 2026-06-01 closed the second round of gaps vs rtk 0.37.1 and caveman:
+
+- **`tok.CompressCaveman(text, intensity)`** — caveman-style prompt compression (Lite / Full / Ultra) as a public Go API. ~150 phrase substitutions, drop-lists for articles / filler / pleasantries, and auto-clarity (security/destructive segments pass through verbatim). `intensity` is monotonic: `len(ultra) <= len(full) <= len(lite)`.
+- **`tok.IsSensitiveFilename(path)`** — 3-layer filename detection (exact basename, sensitive directory, name token). Companion to the content-based `SecretDetector`. Catches `.env`, `id_rsa`, `~/.ssh/...`, `test_credentials.json`, etc. Returns the category that fired.
+- **`tok.SmartTruncate(content, maxLines, lang)`** — code truncation that preserves function signatures and **always reports the exact drop count** in the marker (`kept + dropped == total`, rtk invariant).
+- **`tok.ExtractJSON(text)` / `tok.ExtractJSONArray(text)` / `tok.ExtractAllJSON(text)`** — brace-balanced JSON extraction that handles LLM output with surrounding prose, markdown code fences, and unterminated objects. Apostrophes in English prose are not confused with string delimiters.
+- **`tok.NewTracker(ctx)`** — persistent gain tracker (SQLite + WAL, 90-day retention, pure-Go via `modernc.org/sqlite`). Records per-event savings and supports `Aggregate`, `Recent`, `Prune` queries. Default path `~/.tok/tracker.db`.
+- **`tok.EstimateTokensFast/WithEncoding/ForModel`** — model-aware token estimation exposed at the top level (was previously internal).
+- **`filter.CompressWithRetry`** — validate-fix-retry loop: caller supplies a `Validator` and `AdjustFunc`; the loop escalates mode/intensity and retries up to N times.
+- **`filter.NewTOMLFilter` / `LoadTOMLFilterFile`** — full rtk 8-stage pipeline (strip_ansi, replace regex, match_output short-circuit, strip/keep_lines, truncate_lines, head/tail, max_lines, on_empty) as a `Filter` implementation, pluggable into the main pipeline.
+
+### Earlier (2026-04-20)
+
 Session 2026-04-20 closed the last gaps vs rtk 0.37.1 and caveman:
 
 - **`tok commit-msg`** — read staged diff, emit Conventional Commits subject. Rule-based, no LLM.