Commit 976c558
committed
docs(VisualBuilderPlan): F-G expanded to Universal Tokenizer Playground
Per user feedback: not just our non-standard tokenizer, but ANY
tokenizer with side-loading + multi-tokenizer side-by-side compare.
§9.5 rewritten with new structure:
- 9.5.1 Tokenizer loader — 5 sources: preset library (12+ built-in:
GPT-4/3.5/2, Claude, Llama-3, Mistral, Gemma, Qwen, DeepSeek-V3,
Phi-3, cppmega_v3, nanochat_v3), HuggingFace Hub URL, local file
upload (.json/.tiktoken/.model), paste JSON, custom tiktoken
encoding (BPE ranks + specials + regex)
- 9.5.2 Multi-format runtime registry with lazy-load:
* HF tokenizer.json → @huggingface/tokenizers (8.3 kB, default)
* tiktoken → tiktoken/lite WASM (~300-700 KB, lazy)
* SentencePiece → @sctg/sentencepiece-js WASM (~600 KB, lazy)
* Multi-provider → @goliapkg/tiktoken-wasm (~3 MB, opt-in)
JupyterLite payload stays <1MB until WASM-heavy preset triggered
- 9.5.3 Side-by-side A/B/C compare (up to 3 tokenizers):
same input, per-row chips, cross-panel highlight bus (exBERT
pattern — not in stock playgrounds), diff metrics dashboard
(count delta / Venn / bytes-per-token / cost)
- 9.5.4 Tokenizer-aware special-token classifier:
FIM (red), CODE (blue), CONVERSATION (violet), DOC_SEP (green),
MTP (amber), Engram (cyan), whitespace-sentinel (yellow lossy),
generic special (grey fallback) — applied to ANY tokenizer with
per-tokenizer overrides
- 9.5.5 Greenfield layers (~800 LoC, up from ~600)
- 9.5.6 New API endpoint with primary + compare_with[] + diff_metrics
- 9.5.7 UI layout: A/B/C tokenizer pickers, three-column token chip
panels, diff metrics row, doc-packing strip (primary only)
F-G ticket (cppmega-mlx-o0k.7) updated with same scope.
Budget: F-G 2 weeks (up from 1-1.5); total effort 10 weeks (up from 9.5);
wall-clock still ~8 weeks (F-G parallelises with F-C/F-D).1 parent 13d9514 commit 976c558
1 file changed
Lines changed: 188 additions & 116 deletions
0 commit comments