docs(CLAUDE.md): clarify project positioning and architecture overview

Refactor the documentation to better explain Vectorless as having
two distinct layers: the Rust standard (IR + navigation primitives)
and the Python reference implementation. Update project structure
description with clearer crate categorization and dependency layers.
Revise development workflow guidelines to emphasize the contract
between the standard and reference implementation.

Author: zTgx

CLAUDE.md

@@ -1,53 +1,55 @@
 # CLAUDE.md
 
-Vectorless is a Document Understanding Engine for AI. Compile pipeline is written in Rust; the reasoning/retrieval (ask) layer is written in Python on top of LLM tool-use. See `README.md` for the project's public positioning.
+Vectorless is a Document Understanding Engine for AI. See `README.md` for the project's public positioning.
 
-## Project Structure
+## Positioning: a standard + a reference implementation
 
-Cargo workspace with **13 Rust crates** (compile pipeline + bindings) plus a **Python package** (`vectorless/`) that owns the ask/reasoning loop.
+Vectorless is split into two layers that are **independently consumable**:
 
-### Rust crates (`crates/`)
+1. **The Rust side defines two standards.**
+   - **The IR** (`vectorless-document::Document`) — a single, versioned, serializable artifact that fully represents an understood document. Produced by the compile pipeline; can be persisted, transmitted, and re-loaded.
+   - **The navigation primitives** (`vectorless-primitives::DocumentNavigator`) — a fixed set of shell-style operations (`ls`, `cd`, `cat`, `grep`, `find`, `head`, `wc`, `pwd`, `back`, plus inspection and reasoning-index queries) that any agent uses to read an IR. Exposed to Python via PyO3.
 
-```
-crates/
-├── vectorless-error/        # Error types (Result, Error enum)
-├── vectorless-document/     # Document types (Document, Tree, NavigationIndex)
-├── vectorless-config/       # Configuration hub (aggregates all config types)
-├── vectorless-utils/        # Utilities (fingerprinting, token counting, keyword extraction)
-├── vectorless-graph/        # Cross-document relationship graph
-├── vectorless-events/       # Event system for progress monitoring
-├── vectorless-metrics/      # Metrics collection and reporting
-├── vectorless-llm/          # LLM client (pool, memo/cache, throttle)
-├── vectorless-storage/      # Persistence (Workspace, LRU cache, file backend)
-├── vectorless-compiler/     # Compile pipeline (frontend → transform → analysis → backend)
-├── vectorless-primitives/   # Document navigation primitives (DocumentNavigator)
-├── vectorless-engine/       # Facade (Engine, EngineBuilder) — re-exports public API
-└── vectorless-py/           # PyO3 bindings (compiled into Python native module)
-```
+   These two together are **the standard**. Anything that produces a valid `Document` and consumes it via `DocumentNavigator` is a conforming participant.
 
-### Python package (`vectorless/`)
+2. **The Python side is one reference agent implementation.** `vectorless/ask/` is a multi-agent system (Orchestrator + Workers + reasoning + verify) that ships in the box. It is the default `engine.ask()` — but it is not load-bearing for the standard. Developers can:
+   - Replace `ask/` entirely with their own Python agent built on the same `Document` / `DocumentNavigator` primitives.
+   - Skip Python and consume the IR + primitives directly from Rust.
+   - Produce IRs from a non-Rust source (the IR is serializable JSON) and feed them into any compatible navigator/agent.
+
+The reason this matters: the value of Vectorless is the **IR + primitives contract**, not the bundled reasoning loop. The reference implementation is a courtesy, not a lock-in.
+
+## Project Structure
 
 ```
-vectorless/
-├── ask/                     # Reasoning loop (Orchestrator + Workers + supervisor)
-│   ├── orchestrator.py      # Top-level coordinator
-│   ├── worker/              # Navigation Worker (ls/cd/cat/grep/find/head/wc/chain)
-│   ├── dispatcher.py        # Worker dispatch
-│   ├── plan.py              # Replanning
-│   ├── understand.py        # Query understanding → QueryPlan
-│   ├── reasoning/           # Reasoning chain analyzer
-│   ├── evaluate.py          # Evidence sufficiency evaluation
-│   ├── verify/              # Answer verifier
-│   ├── blackboard.py        # Shared evidence state
-│   └── prompts.py
-├── rerank/                  # Dedup + quality scoring + synthesis
-├── _internal/               # Wrappers around the PyO3 native module
-├── engine.py                # User-facing Engine class
-└── cli/                     # CLI entrypoint
+crates/                       Rust — the standard
+├── vectorless-error/         Result / Error
+├── vectorless-document/      ★ IR types: Document, DocumentTree, NavigationIndex, ReasoningIndex,
+│                              QueryRoutingTable, ChainIndex, ContentOverlapMap, EvidenceScoreMap
+├── vectorless-primitives/    ★ DocumentNavigator (the navigation API)
+├── vectorless-config/        Configuration hub
+├── vectorless-utils/         Fingerprinting, token counting, keyword extraction
+├── vectorless-graph/         Cross-document relationship graph
+├── vectorless-events/        Progress event bus
+├── vectorless-metrics/       Metrics collection
+├── vectorless-llm/           LLM client (pool, throttle, retry)
+├── vectorless-storage/       Workspace persistence (file backend, LRU cache)
+├── vectorless-compiler/      Compile pipeline (frontend → transform → analysis → backend)
+├── vectorless-engine/        Facade: Engine, EngineBuilder, compile() / forget() / load_document()
+└── vectorless-py/            PyO3 bindings → produces the `_vectorless` native module
+
+vectorless/                   Python — the reference multi-agent implementation
+├── ask/                      Orchestrator + Worker + reasoning + verify (the default agent)
+├── rerank/                   Dedup + quality scoring + synthesis
+├── _internal/                Wrappers around the PyO3 native module
+├── engine.py                 User-facing Engine (delegates compile to Rust, ask to ask/)
+└── cli/                      CLI entrypoint
+
+examples/                     Python examples
+docs/                         Docusaurus documentation site
 ```
 
-- `examples/` — Python examples (primary, for Python ecosystem)
-- `docs/` — Docusaurus documentation site
+The two crates marked ★ are the **standard surface**. Everything else in `crates/` is implementation detail that produces and serves that standard.
 
 ### Dependency Layers (Rust)
 
@@ -58,47 +60,51 @@ Layer 2:  config · metrics                              (depends on Layer 0–1
 Layer 3:  llm · storage · primitives                    (depends on Layer 0–2)
 Layer 4:  compiler                                      (depends on Layer 0–3)
 Layer 5:  engine                                        (depends on Layer 0–4)
-Layer 6:  vectorless-py (PyO3 bindings)                 (depends on engine + primitives)
+Layer 6:  vectorless-py (PyO3 bindings)                 (engine + document + primitives)
 ```
 
-### Compile Pipeline
+`vectorless-document` and `vectorless-primitives` have no dependency on `compiler`, `llm`, `storage`, or `engine` — by design, so the standard can be consumed without dragging in the pipeline.
+
+### Compile Pipeline (the IR producer)
 
 The compiler runs documents through four stage groups (`crates/vectorless-compiler/src/passes/`):
 
 ```
-frontend  (parse, build)              ← raw bytes → DocumentTree
-transform (split, enrich)             ← chunking + section enrichment
-analysis  (validate, enhance)         ← structure checks + augmentation
+frontend  (parse, build)              raw bytes → DocumentTree
+transform (split, enrich)             chunking + section enrichment
+analysis  (validate, enhance)         structure checks + augmentation
 backend   (route, concept, navigation,
            chain, overlap, reasoning,
-           score, optimize, verify)   ← retrieval-acceleration artifacts
+           score, optimize, verify)   retrieval-acceleration artifacts
 ```
 
+Every backend pass attaches an optional acceleration field to the `Document`. The IR is valid without any of them (an agent can navigate using only `tree` + `nav_index`); the acceleration fields exist so well-equipped agents can short-circuit reasoning.
+
 ### Compilation Isolation
 
 改一个模块只重编译该 crate + 上游 facade：
+
 - 改 `llm` → llm, compiler, engine, py 重编译；storage/graph 不动
 - 改 `compiler` → compiler, engine, py 重编译；llm/storage 不动
-- 改 `document` → 全部重编译（核心类型，预期行为）
-- 改 Python `ask/`、`rerank/` → 不触发 Rust 重编译
+- 改 `document` 或 `primitives` → 全部重编译（标准变更，预期行为）
+- 改 Python `ask/` / `rerank/` → 不触发 Rust 重编译
 
-### Ask Call Flow
+### How third parties consume the standard
 
-```
-engine.ask()  [Python]
-  → ask/understand()        → QueryPlan (intent + concepts + strategy, LLM-driven)
-  → ask/orchestrator
-      ├── analyze(QueryPlan)            → dispatch plan
-      └── supervisor loop:
-          dispatch Workers (Rust DocumentNavigator via PyO3)
-            → execute nav commands (ls/cd/cat/grep/find/head/wc/chain)
-          → evaluate(blackboard)        → sufficiency check
-          → if insufficient: plan.replan() → loop
-      → rerank/ (dedup → quality score → synthesize)
-  → verify/                  → final answer check
+Anyone building a custom agent works against two types and two crates only:
+
+```rust
+use vectorless_document::Document;             // The IR
+use vectorless_primitives::DocumentNavigator;  // The primitive API
+
+let doc: Document = load_ir_from_disk()?;       // produced by our compiler, or yours
+let mut nav = DocumentNavigator::new(doc);
+let children = nav.ls().await;
+nav.cd("n3").await?;
+let body = nav.cat(None).await?;
 ```
 
-The Rust side exposes `DocumentNavigator` (`vectorless-primitives`) and compiled artifacts; the Python orchestrator drives the LLM reasoning loop and calls back into Rust for fast document operations.
+The Python reference agent (`vectorless/ask/`) is just one consumer of this same surface.
 
 ## Build Commands
 
@@ -107,7 +113,7 @@ The Rust side exposes `DocumentNavigator` (`vectorless-primitives`) and compiled
 cargo build          # Build all crates
 cargo test           # Run workspace tests
 cargo clippy         # Lint
-cargo fmt            # Format code
+cargo fmt            # Format
 
 # Build a single crate (fast — only that crate + dependents)
 cargo build -p vectorless-compiler
@@ -199,10 +205,10 @@ When uncertain whether an operation is safe, **default to asking user confirmati
 
 ## Common Development Workflow
 
-1. **Adding compile-pipeline features**: implement under `crates/vectorless-compiler/src/passes/` (frontend/transform/analysis/backend), add tests in the same module.
-2. **Adding reasoning/ask features**: implement under `vectorless/ask/` (Python), add prompts in `prompts.py`.
-3. **Fixing bugs**: write a failing test first, then fix.
-4. **Adding crates**: new modules get their own crate under `crates/`, registered in workspace `Cargo.toml`.
-5. **Python bindings**: update `crates/vectorless-py/src/lib.rs` (PyO3) when Rust APIs cross the FFI boundary; corresponding wrappers go in `vectorless/_internal/`.
-6. **Python SDK surface**: update `vectorless/engine.py` and related modules when the public API changes.
-7. **Committing**: use Conventional Commits — `feat(compiler): ...`, `fix(ask): ...`, `refactor(engine): ...`.
+1. **Touching the standard (`document` or `primitives`):** this is a public-contract change. Bump `CURRENT_SCHEMA_VERSION` if the IR's serialized shape changes; document the new field; add tests.
+2. **Adding a compile-pipeline pass:** implement under `crates/vectorless-compiler/src/passes/` (frontend/transform/analysis/backend), wire into the pipeline, attach output to `Document` as an `Option<...>` so old IRs still load.
+3. **Adding/modifying reasoning behavior:** implement under `vectorless/ask/` (Python). This is reference-implementation work, not standard work — it should not require Rust changes.
+4. **Fixing bugs:** write a failing test first, then fix.
+5. **Adding crates:** new modules get their own crate under `crates/`, registered in workspace `Cargo.toml`.
+6. **PyO3 bindings:** update `crates/vectorless-py/src/lib.rs` when Rust types cross the FFI boundary; corresponding Python wrappers in `vectorless/_internal/`.
+7. **Committing:** Conventional Commits — `feat(compiler): ...`, `fix(ask): ...`, `refactor(primitives): ...`.