docs(nook): scope tree-sitter highlighting feasibility (roadmap item 3)

Author: truffle-dev

cmd/nook/ROADMAP.md

@@ -71,8 +71,15 @@ Roughly in priority order.
    product decision before build — it is a large surface and should not
    land half-done.
 3. **Tree-sitter highlighting.** Upgrade the syntax backend from chroma
-   to tree-sitter (via WASM) for incremental, error-tolerant parsing.
-   chroma stays the fallback for grammars not yet wired.
+   to tree-sitter for incremental, error-tolerant parsing. chroma stays
+   the fallback for grammars not yet wired. Feasibility is scoped in
+   `docs/nook/design/02-tree-sitter-highlighting.md`: a CGO-free pure-Go
+   runtime (odvcencio/gotreesitter) highlights correctly and reparses
+   incrementally at ~69µs/snippet, but importing its grammar umbrella
+   bloats the binary to 31M (init-time registration defeats DCE). The
+   design pivot is to feed the runtime only the grammar blobs nook wires;
+   the first slice proves that keeps the binary lean before any host
+   wiring lands.
 4. **Minimap.** Optional document overview gutter. Lowest priority;
    useful but not load-bearing for the editing experience. The geometry
    foundation is in place: `internal/minimap` is a pure, constant-time

docs/nook/design/02-tree-sitter-highlighting.md

@@ -0,0 +1,109 @@
+# Tree-sitter highlighting: feasibility and shape
+
+Status: design, backed by a throwaway spike. Roadmap item 3. Today nook
+highlights with chroma (regex/lexer based). The goal is incremental,
+error-tolerant parsing so highlights stay correct mid-edit and so the
+same trees can later feed folding, structural selection, and tags.
+chroma stays the fallback for grammars we do not wire.
+
+## The hard constraint the choice has to respect
+
+nook ships as a single binary installed with
+`go install github.com/truffle-dev/glyph/cmd/nook@latest`, and its north
+star is the lightest, fastest terminal IDE. Two consequences gate the
+backend before any API taste question:
+
+- No C toolchain at install time. That rules out the CGO bindings
+  (`smacker/go-tree-sitter`, `tree-sitter/go-tree-sitter`): they need
+  `CGO_ENABLED=1`, a C compiler, and the grammar C sources on the user's
+  machine. A CGO dependency turns `go install` into "install a compiler
+  first," which is a regression we will not ship.
+- Binary size is a feature. nook is ~5.6M today. A backend that adds tens
+  of megabytes fails the north star even if it works.
+
+That leaves two CGO-free families, both pure Go:
+
+1. `github.com/odvcencio/gotreesitter` — a pure-Go reimplementation of the
+   tree-sitter runtime (parser, lexer, query engine, incremental reparse),
+   200+ grammars, built-in highlight queries. Cross-compiles anywhere Go
+   does, including wasip1.
+2. `github.com/malivvan/tree-sitter` — the upstream C tree-sitter compiled
+   to WASM, run under `wazero` (pure-Go WASM runtime). You embed only the
+   grammar `.wasm` blobs you ship.
+
+## What the spike measured (gotreesitter v0.20.9)
+
+A standalone module, built with `CGO_ENABLED=0`, exercised the full path
+for a Go source snippet:
+
+- `grammars.DetectLanguage("main.go")` resolves the language from the
+  filename; `entry.Language()` is the lazy grammar loader and
+  `entry.HighlightQuery` is the bundled `.scm`.
+- `ts.NewHighlighter(lang, entry.HighlightQuery).Highlight(src)` returned
+  14 `HighlightRange{StartByte, EndByte, Capture}` spans with correct
+  captures — `keyword`, `string`, `comment`, `type`, `property`,
+  `variable`. Capture names map cleanly onto nook's existing highlight
+  style classes, the same seam chroma feeds today.
+- `HighlightIncremental(src, oldTree)` reparsed after an edit and returned
+  an updated span set. Incremental reparse is the whole reason to leave
+  chroma, and it works.
+- Latency: highlighting a small function was ~69µs. Well inside a
+  first-paint budget for a screenful; nothing here blocks a frame.
+
+So the runtime is correct, CGO-free, and fast. The problem is size.
+
+## The size finding, and the pivot it forces
+
+Importing `github.com/odvcencio/gotreesitter/grammars` produced a **31M**
+binary — a 6x bloat over nook's ~5.6M — and it stayed 31M even when the
+program referenced only `grammars.GoLanguage()`. Go's dead-code
+elimination cannot help: the grammars package registers every language
+through `Register(LangEntry{...})` calls at package `init()`
+(`registry_builtin_gen.go`, 542 init-bearing files), and init side
+effects are never eliminated. Importing the umbrella for one language
+costs all 200.
+
+That kills the naive "import the grammars package" approach for nook. It
+does not kill gotreesitter. The runtime exposes the escape hatch directly:
+`ts.LoadLanguage(data []byte)` loads a grammar from a serialized blob, and
+the grammars tree carries `blob_source_external*.go` and
+`blob_source_subset_embedded.go` — i.e. the runtime already supports
+loading grammar blobs from disk and embedding a chosen subset rather than
+the whole registry. The design pivot is to bypass the umbrella package and
+feed the runtime only the grammar blobs nook wires.
+
+## Proposed first slice (green, size-gated)
+
+Prove the lean path before building any host wiring:
+
+1. A tiny spike that embeds exactly one grammar blob (Go) via the
+   subset/external blob source and calls `ts.LoadLanguage` +
+   `NewHighlighter`, with **no import of the umbrella `grammars`
+   package**. Success criterion is a binary within a small delta of
+   baseline (target: single-digit MB add, not 29M) that still highlights
+   Go correctly. If subset-embed cannot escape the registry init, the blob
+   loads from disk at runtime instead (tree-sitter becomes opt-in, grammar
+   blobs shipped beside the binary or fetched on first use).
+2. If neither keeps the binary lean, fall back to the WASM/wazero family
+   (`malivvan/tree-sitter`): embed only wired `.wasm` grammar blobs, pay
+   the wazero runtime once. Re-measure size and latency the same way.
+
+Only after one of those keeps nook light do we touch the host: a
+`tshl` package that wraps the chosen backend behind the same
+capture-name → style-class contract chroma uses today, wired as the
+primary highlighter with chroma as the fallback for unwired languages.
+Nothing here changes the render path until the size question is settled,
+so the first-paint rule and the binary-size north star both hold.
+
+## Open questions for the build hour
+
+- Does subset-embedding one grammar actually avoid the 542 init
+  registrations, or is the registry all-or-nothing? (Determines slice 1
+  vs. runtime-blob-loading vs. WASM.)
+- Is upstream willing to expose per-grammar subpackages / DCE-friendly
+  registration? A lean single-language import would make this trivial and
+  is worth an issue against odvcencio/gotreesitter once the spike pins the
+  exact constraint.
+- Highlight query dialect: gotreesitter ships `.scm` queries per language;
+  confirm they match nook's expected capture vocabulary or add a small
+  capture-name remap in `tshl`.