docs: add Antora documentation site following Divio standard

AsciiDoc documentation in Antora site format with four Divio categories:

Tutorials:
- Getting started with the skainet CLI
- Tool calling with any model via ChatSession
- Running smoke tests

How-to Guides:
- Add a new model architecture (DSL vs hand-coded)
- Add a compute backend
- Add a custom tool
- Use the unified CLI

Reference:
- Architecture overview and module structure
- Inference pipeline stages
- Tokenizer API and TokenizerFactory
- ChatSession API
- Model Registry and UnifiedModelLoader
- CLI reference (skainet + model-specific CLIs)

Explanation:
- Pipeline design decisions (why stages are separated)
- DSL networks vs hand-coded runtimes (trade-offs)
- Tokenizer internals (SentencePiece, BPE, WordPiece)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: michalharakal

docs/antora.yml

@@ -0,0 +1,5 @@
+name: skainet-transformers
+title: SKaiNET Transformers
+version: ~
+nav:
+  - modules/ROOT/nav.adoc

docs/modules/ROOT/nav.adoc

@@ -0,0 +1,25 @@
+* xref:index.adoc[Overview]
+
+.Tutorials
+* xref:tutorials/getting-started.adoc[Getting Started]
+* xref:tutorials/tool-calling.adoc[Tool Calling with Any Model]
+* xref:tutorials/smoke-tests.adoc[Running Smoke Tests]
+
+.How-to Guides
+* xref:how-to/add-model.adoc[Add a New Model Architecture]
+* xref:how-to/add-compute-backend.adoc[Add a Compute Backend]
+* xref:how-to/add-tool.adoc[Add a Custom Tool]
+* xref:how-to/run-unified-cli.adoc[Use the Unified CLI]
+
+.Reference
+* xref:reference/architecture.adoc[Architecture Overview]
+* xref:reference/pipeline.adoc[Inference Pipeline]
+* xref:reference/tokenizer-api.adoc[Tokenizer API]
+* xref:reference/chat-session-api.adoc[ChatSession API]
+* xref:reference/model-registry.adoc[Model Registry]
+* xref:reference/cli-reference.adoc[CLI Reference]
+
+.Explanation
+* xref:explanation/pipeline-design.adoc[Pipeline Design Decisions]
+* xref:explanation/dsl-vs-handcoded.adoc[DSL Networks vs Hand-Coded Runtimes]
+* xref:explanation/tokenizer-internals.adoc[Tokenizer Internals]

docs/modules/ROOT/pages/explanation/dsl-vs-handcoded.adoc

@@ -0,0 +1,124 @@
+= DSL Networks vs Hand-Coded Runtimes
+:description: Why DSL network definitions replace hand-coded runtimes.
+
+== Two Approaches to Model Definition
+
+SKaiNET Transformers supports two ways to define a model's forward pass:
+
+=== Hand-Coded Runtime (Legacy)
+
+A class that extends `DecoderRuntime` and implements each layer explicitly:
+
+[source,kotlin]
+----
+class LlamaRuntime<T>(/* ... */) : DecoderRuntime<T>(ctx, dtype) {
+    override fun runLayer(layerIdx: Int, x: Tensor<T, Float>): Tensor<T, Float> {
+        val normed = rmsNorm(x, weights.attnNorm[layerIdx])
+        val q = matmul(normed, weights.wq[layerIdx])
+        val k = matmul(normed, weights.wk[layerIdx])
+        // ... 50+ lines of attention + FFN
+    }
+}
+----
+
+=== DSL Network Definition (Current)
+
+A pure function that declares the architecture using the network DSL:
+
+[source,kotlin]
+----
+fun <T : DType, V> llamaNetwork(metadata: LlamaModelMetadata): Module<T, V> {
+    return sequential<T, V> {
+        embedding(vocabSize, dim, id = "token_embd")
+        for (layer in 0 until nLayers) {
+            rmsNorm(dim, eps, id = "attn_norm")
+            multiHeadAttention(dim, nHeads, nKVHeads, causal = true) {
+                rope(headDim, seqLen)
+                kvCache(seqLen, nKVHeads, headDim)
+            }
+            residual()
+            rmsNorm(dim, eps, id = "ffn_norm")
+            swiGluFFN(dim, ffnDim)
+            residual()
+        }
+        rmsNorm(dim, eps, id = "output_norm")
+    }
+}
+----
+
+== Why DSL is Preferred
+
+=== Compute Graph Optimization
+
+DSL networks can be traced into a ComputeGraph (DAG) and optimized:
+
+* *TransposeEliminationPass* -- folds weight transposes into matmul, eliminating O(n^2) copies
+* *LLMFusionPass* -- fuses RMSNorm (7 ops -> 1), SwiGLU FFN (5 ops -> 1), QKV projections (3 -> 1)
+* *DeadCodeEliminationPass* -- removes unused intermediate tensors
+
+Hand-coded runtimes cannot benefit from these optimizations because operations are imperative, not declarative.
+
+=== Weight Loading is Automatic
+
+DSL modules have named parameters (e.g., `"blk.0/attn/q_proj"`).
+`WeightMapper` matches these to GGUF tensor names via `WeightNameResolver`.
+No manual weight loading code needed.
+
+=== Multiple Execution Modes
+
+The same DSL definition supports:
+
+`DIRECT`:: Execute the module tree directly (debugging, correctness testing)
+`HYBRID`:: Compile compute-heavy subgraphs, run attention imperatively (best balance)
+`OPTIMIZED`:: Full DAG compilation and execution (maximum performance)
+
+=== Adding New Architectures is Simpler
+
+A new architecture is a single function, not a 500-line class.
+If the architecture uses standard building blocks (MHA, RMSNorm, FFN), the DSL already has them.
+
+== When Hand-Coded Runtimes Are Needed
+
+Some architectures have components the DSL cannot express:
+
+* *Qwen3.5 DeltaNet* -- hybrid DeltaNet (linear attention + SSM) layers with causal 1D convolution
+* *Gemma3n* -- variable FFN dimensions per layer (MatFormer), per-layer embeddings
+* *Voxtral* -- ODE flow matching for audio codec
+
+These use `DecoderRuntime` directly.
+The goal is to extend the DSL to support these patterns over time.
+
+== Current Status
+
+[cols="1,1,1"]
+|===
+|Model |DSL |Status
+
+|LLaMA/Mistral
+|`llamaNetwork()`
+|`LlamaRuntime` deprecated
+
+|Qwen2/3
+|`qwenNetwork()`
+|Delegates to `llamaNetwork()`
+
+|Apertus
+|`apertusNetwork()`
+|`ApertusRuntime` deprecated
+
+|BERT
+|`bertNetwork()`
+|`BertRuntime` deprecated
+
+|Voxtral
+|`voxtralBackboneNetwork()`
+|Partial DSL
+
+|Gemma3n
+|_none_
+|Hand-coded only
+
+|Qwen3.5
+|_none_
+|Hand-coded (DeltaNet)
+|===

docs/modules/ROOT/pages/explanation/pipeline-design.adoc

@@ -0,0 +1,47 @@
+= Pipeline Design Decisions
+:description: Why the inference pipeline is structured as separate stages.
+
+== The Problem
+
+Early SKaiNET had a monolithic approach: each model family (LLaMA, Gemma, Apertus) had its own hand-coded runtime that handled everything -- weight loading, forward pass, KV cache, tokenization, and generation.
+This led to:
+
+* *Duplicated logic* -- each runtime reimplemented `generate()`, `sample()`, `forward()`.
+* *Tight coupling* -- tool calling only worked with kllama because `ToolCallingDemo` depended on `GGUFTokenizer`, a kllama-specific class.
+* *No optimization* -- hand-coded runtimes couldn't benefit from compute graph optimization passes.
+
+== The Solution: Separated Pipeline Stages
+
+The pipeline is split into stages that are independently replaceable:
+
+[horizontal]
+Weight Loading:: Parse GGUF/SafeTensors into typed tensor maps. Model-format concern, not architecture concern.
+Network Definition:: Pure functions (`llamaNetwork()`, `apertusNetwork()`) that return a `Module` tree. Architecture concern only.
+Graph Compilation:: Trace the module tree into a DAG, apply optimization passes. Framework concern.
+Inference Runtime:: `forward(tokenId)` and `generate()`. Pure inference, no I/O.
+Tokenization:: `encode()`/`decode()`. Completely independent of model architecture.
+Chat Pipeline:: `ChatSession`, `AgentLoop`, `ChatTemplate`. Independent of both model and tokenizer implementation.
+
+== Key Design Decisions
+
+=== Tokenizer Interface with Metadata
+
+The `Tokenizer` interface includes `eosTokenId`, `bosTokenId`, and `vocabSize`.
+This eliminated the need for the `GGUFTokenizer` downcast that previously coupled tool calling to kllama.
+Any tokenizer implementation works with `ChatSession` and `AgentLoop`.
+
+=== ChatSession as the Composition Root
+
+Rather than having each CLI wire up `InferenceRuntime` + `Tokenizer` + `ChatTemplate` + `ToolRegistry` individually, `ChatSession` bundles them.
+A runner creates one `ChatSession` and gets chat, agent, and demo modes for free.
+
+=== ModelRegistry for Auto-Detection
+
+Instead of if/else chains in each CLI to determine which loader to use, `ModelRegistry.detect(architecture)` returns a `ModelFamily` enum with capabilities (tool calling support, chat template family).
+The unified `skainet` CLI uses this to load any GGUF model without architecture-specific flags.
+
+=== GGUFTokenizer in llm-core
+
+Moving `GGUFTokenizer` from `kllama` to `llm-core` was essential.
+Every runner needed it, but depending on `kllama` just for the tokenizer created circular dependency pressure.
+The `TokenizerFactory` in `llm-core` provides a clean entry point.

docs/modules/ROOT/pages/explanation/tokenizer-internals.adoc

@@ -0,0 +1,68 @@
+= Tokenizer Internals
+:description: How tokenization works across different model families.
+
+== Tokenizer Types
+
+Different model families use different tokenization strategies.
+The `GGUFTokenizer` auto-detects the type from GGUF metadata.
+
+=== SentencePiece (LLaMA, Gemma)
+
+* Space is encoded as `\u2581` (lower one eighth block)
+* Subword units are learned from training data
+* GGUF metadata field: `tokenizer.ggml.model = "llama"` or `"sentencepiece"`
+
+=== BPE (Qwen, Mistral, GPT-2)
+
+* Byte-level BPE: text is converted to UTF-8 bytes, each byte mapped to a Unicode character
+* The byte-to-Unicode mapping avoids control characters (bytes 0-32 map to U+0100+)
+* Space is represented as `\u0120` (Latin capital G with dot above)
+* GGUF metadata field: `tokenizer.ggml.model = "gpt2"` or `"bpe"`
+
+=== WordPiece (BERT)
+
+* Subwords prefixed with `##` (e.g., "playing" -> `["play", "##ing"]`)
+* Uses `[CLS]`, `[SEP]`, `[UNK]`, `[PAD]` special tokens
+
+== Special Token Handling
+
+Chat templates use special tokens like `<|im_start|>` and `<|im_end|>` to delimit messages.
+These must be encoded as single tokens, not character-split.
+
+The `GGUFTokenizer` collects special tokens from the vocabulary (tokens matching `<|...|>`) and splits text around them before applying BPE.
+This ensures `<|im_start|>system` encodes as `[151644, 8948]` (two tokens), not as individual characters.
+
+== GGUF Tokenizer Fields
+
+[cols="2,3"]
+|===
+|Field |Description
+
+|`tokenizer.ggml.model`
+|Tokenizer type: `"llama"`, `"gpt2"`, `"bert"`
+
+|`tokenizer.ggml.tokens`
+|Vocabulary as string array
+
+|`tokenizer.ggml.scores`
+|BPE merge scores (SentencePiece)
+
+|`tokenizer.ggml.merges`
+|BPE merge pairs (GPT-2 style)
+
+|`tokenizer.ggml.bos_token_id`
+|Beginning-of-sequence token ID
+
+|`tokenizer.ggml.eos_token_id`
+|End-of-sequence token ID
+
+|`tokenizer.ggml.token_type`
+|Per-token type (normal, control, unknown, byte)
+|===
+
+== TokenizerFactory
+
+`TokenizerFactory` in `llm-core` provides a unified entry point.
+It delegates to `GGUFTokenizer` or `HuggingFaceBPETokenizer` based on the source format.
+
+The factory is the recommended way to create tokenizers -- callers don't need to know which implementation is used.

docs/modules/ROOT/pages/how-to/add-compute-backend.adoc

@@ -0,0 +1,59 @@
+= Add a Compute Backend
+:description: Implement and register a new compute backend (Metal, CUDA, etc.).
+
+Compute backends provide tensor operations for a specific hardware target.
+The system auto-selects the best available backend at startup.
+
+== 1. Implement BackendProvider
+
+[source,kotlin]
+----
+class MetalBackendProvider : BackendProvider {
+    override val name: String = "metal"
+    override val displayName: String = "Apple Metal GPU"
+    override val priority: Int = 100  // higher = preferred
+
+    override fun isAvailable(): Boolean {
+        // Check if Metal is available on this platform
+    }
+
+    override fun createOps(): CpuOps {
+        // Return Metal-accelerated tensor operations
+    }
+}
+----
+
+== 2. Register via ServiceLoader (JVM)
+
+Create `META-INF/services/sk.ainet.lang.backend.BackendProvider`:
+
+[source]
+----
+com.example.MetalBackendProvider
+----
+
+== 3. Register Manually (Native/JS/WASM)
+
+[source,kotlin]
+----
+BackendRegistry.register(MetalBackendProvider())
+----
+
+== 4. Verify
+
+[source,bash]
+----
+./gradlew :llm-apps:skainet-cli:run \
+  --args="-m model.gguf --list-backends"
+----
+
+Output:
+
+----
+Available backends:
+  metal        Apple Metal GPU (priority=100, available)
+  cpu          CPU (SIMD) (priority=0, available)
+----
+
+The backend with the highest priority that is available is auto-selected.
+Override with `--backend=cpu` to force a specific backend.