Skip to content

Commit 319d89a

Browse files
committed
test+docs: cover new commands, advertise session additions
Tests (19 new) - internal/commands/core/pr_review_cmd_test.go (4 tests): groupByFile splits and strips prefix, skips malformed findings; sortedKeys orders correctly and handles empty maps - internal/commands/hooks/hook_mode_test.go (15 tests): writeFlag rejects invalid modes and persists with 0600 perms; readFlag rejects oversized files (>64B), symlinks, and unknown modes; clearFlag removes cleanly; resolveDefaultMode env/config/fallback priority; containsAll/Any helpers evals/bench.sh: fix invocation — use `tok compress --mode aggressive` instead of `tok --mode=full` (the latter just printed help). Real numbers now: git log 89%, git diff 99%, ls 99%, find 99% tokens saved README.md: add Benchmarks table with reproducible numbers + Recent additions list (commit-msg, review-diff, pr-review, md, cheatsheet, hook mode, wenyan filter, release automation, skill bundler, e2e) Pre-session source finalization (unrelated to new features, but ready): - hooks/README.md, hooks/tok-init.sh: drop lingering rtk comparisons - hooks/install.sh: +53 lines of install script - internal/filter/pipeline_process.go: 251 → 76 lines; six-layer refactor pairs with six_layer_pipeline.go added earlier in the branch All tests pass twice (49/49 packages, 0 fails).
1 parent f0059eb commit 319d89a

8 files changed

Lines changed: 344 additions & 236 deletions

File tree

README.md

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ $ tok npm test
3232
# 200 lines → 3 lines: pass/fail + failures
3333
```
3434

35-
### 3. Transparent Command Rewriting (rtk-style)
35+
### 3. Transparent Command Rewriting
3636

3737
Install the hook once. Every bash command from your AI agent is automatically rewritten:
3838

@@ -44,7 +44,7 @@ Claude sees: 420 tokens → 84 tokens (80% saved)
4444

4545
Claude never knows. You don't type `tok` prefix. It just works.
4646

47-
### 4. Make Agents Talk Tersely (caveman-style)
47+
### 4. Make Agents Talk Tersely
4848

4949
Install agent rules that make AI respond with ~75% fewer output tokens:
5050

@@ -117,6 +117,36 @@ tok session # Adoption across sessions
117117

118118
---
119119

120+
## Benchmarks
121+
122+
Measured on this repo via `evals/bench.sh` (raw vs `tok compress --mode aggressive`):
123+
124+
| fixture | raw bytes | raw tokens | tok bytes | tok tokens | saved |
125+
|-----------|----------:|-----------:|----------:|-----------:|------:|
126+
| git log | 2,873 | 718 | 298 | 74 | 89 % |
127+
| git diff | 385,051 | 96,262 | 1,117 | 279 | 99 % |
128+
| ls -la | 66,341 | 16,585 | 148 | 37 | 99 % |
129+
| find .go | 19,145 | 4,786 | 147 | 36 | 99 % |
130+
131+
Reproduce: `go build -o tok ./cmd/tok && TOK=./tok evals/bench.sh --no-rtk`
132+
133+
## Recent additions
134+
135+
Session 2026-04-20 closed the last gaps vs rtk 0.37.1 and caveman:
136+
137+
- **`tok commit-msg`** — read staged diff, emit Conventional Commits subject. Rule-based, no LLM.
138+
- **`tok review-diff`** — scan diff, emit one-line review comments (`🔴 bug / 🟡 risk / 🔵 nit`). Rule-based, no LLM.
139+
- **`tok pr-review [--base|--pr]`** — batch `review-diff` across a whole PR, grouped by file.
140+
- **`tok md <file>`** — compress markdown/memory file in place with `.original.md` backup. New wenyan modes.
141+
- **`tok cheatsheet`** — one-shot reference card for shell users (`modes` and `quickref` are aliases).
142+
- **`tok hook mode {activate|track|status|set}`** — Go-native SessionStart + UserPromptSubmit hook bodies. Drop-in replacement for the Node.js scripts, same flag-file format.
143+
- **Wenyan filter layer** (`internal/filter/wenyan.go`) — classical-Chinese-inspired rule-based compression, callable from the main pipeline.
144+
- **Release automation**`release-please` workflow + `Formula/tok.rb` in-repo.
145+
- **Skill bundler**`scripts/build-skill.sh` produces `tok.skill` zip for single-file distribution.
146+
- **End-to-end harness**`tests/e2e/` Docker-based scenario runner.
147+
148+
---
149+
120150
## Features
121151

122152
### Input Compression (6 Modes)

evals/bench.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,9 @@ for f in "${fixtures[@]}"; do
7373
raw_tok=$(est_tokens "$raw")
7474
printf "%-12s %-10s %10d %10d %8s %8d\n" "$label" "raw" "$raw_bytes" "$raw_tok" "-" "$((t1 - t0))"
7575

76-
# tok
76+
# tok (Unix pipe filter — tok compress)
7777
t0=$(time_ms)
78-
"$TOK" --mode=full < "$raw" > "$tok_out" 2>/dev/null || cp "$raw" "$tok_out"
78+
"$TOK" compress --mode aggressive < "$raw" > "$tok_out" 2>/dev/null || cp "$raw" "$tok_out"
7979
t1=$(time_ms)
8080
tok_bytes=$(wc -c < "$tok_out")
8181
tok_tok=$(est_tokens "$tok_out")

hooks/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ powershell -ExecutionPolicy Bypass -File hooks\uninstall.ps1
4040

4141
## Transparent Command Rewriting
4242

43-
The tok transparent rewriting hook works similarly to rtk's approach. It intercepts
43+
The tok transparent rewriting hook intercepts CLI commands. It intercepts
4444
bash commands from AI agent tool calls and rewrites known commands to their tok
4545
equivalents (e.g., `git status``tok git status`). The AI agent never sees the
4646
rewrite - it just gets compressed output.

hooks/install.sh

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,62 @@ with open('$CLAUDE_SETTINGS', 'w') as f:
9090
echo "Configured: Claude Code statusline ($CLAUDE_SETTINGS)"
9191
}
9292

93+
install_js_mode_hooks() {
94+
if ! command -v node >/dev/null 2>&1; then
95+
echo "node not found — skipping JS mode-tracking hooks (install Node.js to enable natural-language activation)"
96+
return
97+
fi
98+
if [[ ! -d "$CLAUDE_CONFIG_DIR" ]]; then
99+
return
100+
fi
101+
102+
local dest="$CLAUDE_CONFIG_DIR/hooks"
103+
mkdir -p "$dest"
104+
cp "$SCRIPT_DIR/tok-mode-config.js" "$dest/tok-mode-config.js"
105+
cp "$SCRIPT_DIR/tok-mode-activate.js" "$dest/tok-mode-activate.js"
106+
cp "$SCRIPT_DIR/tok-mode-tracker.js" "$dest/tok-mode-tracker.js"
107+
chmod 0644 "$dest"/tok-mode-*.js
108+
109+
if command -v python3 >/dev/null 2>&1; then
110+
python3 - "$CLAUDE_SETTINGS" "$dest" <<'PY' 2>/dev/null || echo "Warning: could not merge JS hook settings"
111+
import json, os, sys
112+
settings, dest = sys.argv[1], sys.argv[2]
113+
if os.path.exists(settings):
114+
with open(settings) as f:
115+
cfg = json.load(f)
116+
else:
117+
cfg = {}
118+
hooks = cfg.setdefault('hooks', {})
119+
def has(group, marker):
120+
for entry in hooks.get(group, []):
121+
for h in entry.get('hooks', []):
122+
if marker in h.get('command', ''):
123+
return True
124+
return False
125+
if not has('SessionStart', 'tok-mode-activate.js'):
126+
hooks.setdefault('SessionStart', []).append({
127+
'matcher': 'Always',
128+
'hooks': [{'type': 'command',
129+
'command': f'node "{dest}/tok-mode-activate.js"',
130+
'timeout': 5}],
131+
})
132+
if not has('UserPromptSubmit', 'tok-mode-tracker.js'):
133+
hooks.setdefault('UserPromptSubmit', []).append({
134+
'hooks': [{'type': 'command',
135+
'command': f'node "{dest}/tok-mode-tracker.js"',
136+
'timeout': 5}],
137+
})
138+
with open(settings, 'w') as f:
139+
json.dump(cfg, f, indent=2)
140+
PY
141+
echo "Configured: JS mode-tracking hooks ($dest)"
142+
fi
143+
}
144+
93145
install_in_file "$HOME/.zshrc"
94146
install_in_file "$HOME/.bashrc"
95147
install_claude_code_statusline
148+
install_js_mode_hooks
96149

97150
echo "Done. Restart shell or source your rc file."
98151
echo ""

hooks/tok-init.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# tok-init-version: 1.0
44
#
55
# Sets up the tok rewrite hook for various AI coding assistants.
6-
# Similar to rtk's `rtk init -g` command.
6+
77
#
88
# Usage:
99
# tok init -g # Global install for all agents
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
package core
2+
3+
import (
4+
"strings"
5+
"testing"
6+
)
7+
8+
func TestGroupByFile_SplitsProperly(t *testing.T) {
9+
findings := []string{
10+
"src/a.go:10 🔴 bug hard-coded credential. load from env.",
11+
"src/a.go:22 🟡 risk TODO. resolve.",
12+
"src/b.go:5 🔵 nit console.log. remove.",
13+
}
14+
groups := groupByFile(findings)
15+
if len(groups) != 2 {
16+
t.Fatalf("want 2 files, got %d", len(groups))
17+
}
18+
if len(groups["src/a.go"]) != 2 {
19+
t.Errorf("src/a.go should have 2 findings, got %d", len(groups["src/a.go"]))
20+
}
21+
if len(groups["src/b.go"]) != 1 {
22+
t.Errorf("src/b.go should have 1 finding, got %d", len(groups["src/b.go"]))
23+
}
24+
// Ensure file prefix is stripped — the rest should start with line number.
25+
for _, v := range groups["src/a.go"] {
26+
if strings.HasPrefix(v, "src/a.go") {
27+
t.Errorf("grouped finding still carries file prefix: %q", v)
28+
}
29+
}
30+
}
31+
32+
func TestGroupByFile_SkipsMalformedFindings(t *testing.T) {
33+
findings := []string{
34+
"no-colon-anywhere",
35+
"src/a.go:1 ok",
36+
}
37+
groups := groupByFile(findings)
38+
if len(groups) != 1 {
39+
t.Errorf("want 1 file (malformed skipped), got %d", len(groups))
40+
}
41+
}
42+
43+
func TestSortedKeys_Ordering(t *testing.T) {
44+
m := map[string][]string{
45+
"zeta": {"x"},
46+
"alpha": {"y"},
47+
"mid": {"z"},
48+
}
49+
got := sortedKeys(m)
50+
want := []string{"alpha", "mid", "zeta"}
51+
if len(got) != len(want) {
52+
t.Fatalf("len mismatch: %v vs %v", got, want)
53+
}
54+
for i := range want {
55+
if got[i] != want[i] {
56+
t.Errorf("pos %d: got %q, want %q", i, got[i], want[i])
57+
}
58+
}
59+
}
60+
61+
func TestSortedKeys_EmptyMap(t *testing.T) {
62+
got := sortedKeys(map[string][]string{})
63+
if len(got) != 0 {
64+
t.Errorf("empty map should yield empty slice, got %v", got)
65+
}
66+
}
Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
package hooks
2+
3+
import (
4+
"os"
5+
"path/filepath"
6+
"testing"
7+
)
8+
9+
// testClaudeDir redirects CLAUDE_CONFIG_DIR to a fresh temp dir for the test.
10+
func testClaudeDir(t *testing.T) string {
11+
t.Helper()
12+
dir := t.TempDir()
13+
t.Setenv("CLAUDE_CONFIG_DIR", dir)
14+
return dir
15+
}
16+
17+
func TestWriteFlag_RejectsInvalidMode(t *testing.T) {
18+
testClaudeDir(t)
19+
if err := writeFlag("not-a-mode"); err == nil {
20+
t.Errorf("expected error for invalid mode")
21+
}
22+
}
23+
24+
func TestWriteFlag_PersistsValidMode(t *testing.T) {
25+
dir := testClaudeDir(t)
26+
if err := writeFlag("ultra"); err != nil {
27+
t.Fatalf("writeFlag: %v", err)
28+
}
29+
path := filepath.Join(dir, ".tok-active")
30+
data, err := os.ReadFile(path)
31+
if err != nil {
32+
t.Fatalf("read: %v", err)
33+
}
34+
if string(data) != "ultra" {
35+
t.Errorf("flag body = %q, want ultra", string(data))
36+
}
37+
// Permission check — must be 0600
38+
st, _ := os.Stat(path)
39+
if st.Mode().Perm() != 0o600 {
40+
t.Errorf("flag perms = %o, want 0600", st.Mode().Perm())
41+
}
42+
}
43+
44+
func TestReadFlag_ReturnsEmptyForMissingFile(t *testing.T) {
45+
testClaudeDir(t)
46+
if got := readFlag(); got != "" {
47+
t.Errorf("expected empty, got %q", got)
48+
}
49+
}
50+
51+
func TestReadFlag_RoundTripsValidMode(t *testing.T) {
52+
testClaudeDir(t)
53+
if err := writeFlag("wenyan-full"); err != nil {
54+
t.Fatal(err)
55+
}
56+
if got := readFlag(); got != "wenyan-full" {
57+
t.Errorf("round-trip = %q, want wenyan-full", got)
58+
}
59+
}
60+
61+
func TestReadFlag_RejectsOversizedFile(t *testing.T) {
62+
dir := testClaudeDir(t)
63+
path := filepath.Join(dir, ".tok-active")
64+
// Write a file > 64 bytes
65+
big := make([]byte, maxFlagBytes+10)
66+
for i := range big {
67+
big[i] = 'f'
68+
}
69+
if err := os.MkdirAll(dir, 0o700); err != nil {
70+
t.Fatal(err)
71+
}
72+
if err := os.WriteFile(path, big, 0o600); err != nil {
73+
t.Fatal(err)
74+
}
75+
if got := readFlag(); got != "" {
76+
t.Errorf("oversized flag should yield empty, got %q", got)
77+
}
78+
}
79+
80+
func TestReadFlag_RejectsSymlink(t *testing.T) {
81+
dir := testClaudeDir(t)
82+
target := filepath.Join(dir, "target")
83+
if err := os.WriteFile(target, []byte("ultra"), 0o600); err != nil {
84+
t.Fatal(err)
85+
}
86+
link := filepath.Join(dir, ".tok-active")
87+
if err := os.Symlink(target, link); err != nil {
88+
t.Skip("symlink not supported: " + err.Error())
89+
}
90+
if got := readFlag(); got != "" {
91+
t.Errorf("symlinked flag should yield empty, got %q", got)
92+
}
93+
}
94+
95+
func TestReadFlag_RejectsUnknownMode(t *testing.T) {
96+
dir := testClaudeDir(t)
97+
path := filepath.Join(dir, ".tok-active")
98+
if err := os.WriteFile(path, []byte("super-duper-mode"), 0o600); err != nil {
99+
t.Fatal(err)
100+
}
101+
if got := readFlag(); got != "" {
102+
t.Errorf("unknown mode should yield empty, got %q", got)
103+
}
104+
}
105+
106+
func TestClearFlag_Removes(t *testing.T) {
107+
dir := testClaudeDir(t)
108+
_ = writeFlag("full")
109+
clearFlag()
110+
if _, err := os.Stat(filepath.Join(dir, ".tok-active")); !os.IsNotExist(err) {
111+
t.Errorf("clearFlag did not remove file: %v", err)
112+
}
113+
}
114+
115+
func TestResolveDefaultMode_FallbackFull(t *testing.T) {
116+
t.Setenv("TOK_DEFAULT_MODE", "")
117+
t.Setenv("XDG_CONFIG_HOME", t.TempDir()) // empty config dir
118+
t.Setenv("HOME", t.TempDir())
119+
if got := resolveDefaultMode(); got != "full" {
120+
t.Errorf("fallback = %q, want full", got)
121+
}
122+
}
123+
124+
func TestResolveDefaultMode_EnvWins(t *testing.T) {
125+
t.Setenv("TOK_DEFAULT_MODE", "ULTRA") // case-insensitive
126+
if got := resolveDefaultMode(); got != "ultra" {
127+
t.Errorf("env = %q, want ultra", got)
128+
}
129+
}
130+
131+
func TestResolveDefaultMode_InvalidEnvIgnored(t *testing.T) {
132+
t.Setenv("TOK_DEFAULT_MODE", "nonsense")
133+
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
134+
t.Setenv("HOME", t.TempDir())
135+
if got := resolveDefaultMode(); got != "full" {
136+
t.Errorf("invalid env should fall through, got %q", got)
137+
}
138+
}
139+
140+
func TestResolveDefaultMode_ConfigFile(t *testing.T) {
141+
t.Setenv("TOK_DEFAULT_MODE", "")
142+
xdg := t.TempDir()
143+
t.Setenv("XDG_CONFIG_HOME", xdg)
144+
cfgDir := filepath.Join(xdg, "tok")
145+
_ = os.MkdirAll(cfgDir, 0o700)
146+
_ = os.WriteFile(filepath.Join(cfgDir, "config.json"), []byte(`{"defaultMode":"lite"}`), 0o600)
147+
if got := resolveDefaultMode(); got != "lite" {
148+
t.Errorf("config-file = %q, want lite", got)
149+
}
150+
}
151+
152+
func TestContainsAll(t *testing.T) {
153+
if !containsAll("activate tok please", []string{"tok", "activate"}) {
154+
t.Error("should match both tokens")
155+
}
156+
if containsAll("activate please", []string{"tok", "activate"}) {
157+
t.Error("should not match — tok missing")
158+
}
159+
}
160+
161+
func TestContainsAny(t *testing.T) {
162+
if !containsAny("please turn on now", []string{"enable", "turn on"}) {
163+
t.Error("should match turn on")
164+
}
165+
if containsAny("nope", []string{"enable", "start"}) {
166+
t.Error("should not match")
167+
}
168+
}

0 commit comments

Comments
 (0)