Skip to content

Commit 976c558

Browse files
committed
docs(VisualBuilderPlan): F-G expanded to Universal Tokenizer Playground
Per user feedback: not just our non-standard tokenizer, but ANY tokenizer with side-loading + multi-tokenizer side-by-side compare. §9.5 rewritten with new structure: - 9.5.1 Tokenizer loader — 5 sources: preset library (12+ built-in: GPT-4/3.5/2, Claude, Llama-3, Mistral, Gemma, Qwen, DeepSeek-V3, Phi-3, cppmega_v3, nanochat_v3), HuggingFace Hub URL, local file upload (.json/.tiktoken/.model), paste JSON, custom tiktoken encoding (BPE ranks + specials + regex) - 9.5.2 Multi-format runtime registry with lazy-load: * HF tokenizer.json → @huggingface/tokenizers (8.3 kB, default) * tiktoken → tiktoken/lite WASM (~300-700 KB, lazy) * SentencePiece → @sctg/sentencepiece-js WASM (~600 KB, lazy) * Multi-provider → @goliapkg/tiktoken-wasm (~3 MB, opt-in) JupyterLite payload stays <1MB until WASM-heavy preset triggered - 9.5.3 Side-by-side A/B/C compare (up to 3 tokenizers): same input, per-row chips, cross-panel highlight bus (exBERT pattern — not in stock playgrounds), diff metrics dashboard (count delta / Venn / bytes-per-token / cost) - 9.5.4 Tokenizer-aware special-token classifier: FIM (red), CODE (blue), CONVERSATION (violet), DOC_SEP (green), MTP (amber), Engram (cyan), whitespace-sentinel (yellow lossy), generic special (grey fallback) — applied to ANY tokenizer with per-tokenizer overrides - 9.5.5 Greenfield layers (~800 LoC, up from ~600) - 9.5.6 New API endpoint with primary + compare_with[] + diff_metrics - 9.5.7 UI layout: A/B/C tokenizer pickers, three-column token chip panels, diff metrics row, doc-packing strip (primary only) F-G ticket (cppmega-mlx-o0k.7) updated with same scope. Budget: F-G 2 weeks (up from 1-1.5); total effort 10 weeks (up from 9.5); wall-clock still ~8 weeks (F-G parallelises with F-C/F-D).
1 parent 13d9514 commit 976c558

1 file changed

Lines changed: 188 additions & 116 deletions

File tree

0 commit comments

Comments
 (0)