Merge pull request #195 from AdaWorldAPI/claude/ogar-doc-ir-spider-convergence

docs: doc-layer amendment (A1-A5) + doc-IR × spider-rs convergence plan

Author: AdaWorldAPI

docs/DISCOVERY-MAP.md

@@ -1298,3 +1298,25 @@ isolation. The map's job is to keep them visible.
   Charter: `docs/OGAR-DOC-LAYER-PROPOSAL.md` (merged #191). Grade [S] until the
   5+3 council + a probe promote it; the canon mints (0x080A/0x080B) land WITH
   the council-verified build, never ahead of it.
+
+- **D-DOC-IR-SECOND-RETINA** (2026-07-13; [S] — plan, council gates W1+):
+  `doc.v1` is promoted from OCR output format to the substrate's **perceptual
+  IR** (`ogar-doc-ir`, serde-only neutral tissue), with TWO sanctioned
+  producers: tesseract-rs (pixels) and spider-rs (`AdaWorldAPI/spider` fork,
+  DOM — HTML5 `<header>/<main>/<footer>/<table>` self-labels the regions OCR
+  infers). The code-side pattern replayed on perception: N retinas → one
+  closed doc IR → one awareness subtree, with `ruff_spo_triplet` discipline
+  verbatim (closed region-kind vocab, hard-fail load gate, version marker).
+  Spatial address = u8×u8 unit-square rail (the `X:Y` facet rail; a page IS a
+  256×256 tile — the "2D spatial focus of attention", zero-value-decode
+  region attention). Provenance lane keeps OCR confidence and crawl trust as
+  separate quantities; subtree identity = content sha256 (scan and HTML of
+  the same document converge on ONE `0x080B` subtree). Operator rulings
+  A1–A5 amend the doc-layer charter (one `ogar-doc` crate — split axis is
+  STATE not direction; `DocRenderer` trait with runtime-bound tesseract /
+  Spire.Doc / askama adapters — the one-leg rule; document template =
+  ClassView × WideFieldMask, same brick as Klickwege — no new DSL). Killer
+  falsifier P-XRETINA: same invoice via both retinas ⇒ same typed-field
+  facts + same subtree identity, run BEFORE anything persists. Plan:
+  `docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md`; amendment:
+  `docs/OGAR-DOC-LAYER-PROPOSAL.md` §AMENDMENT. Depends: D-OGAR-DOC-LAYER.

docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md

@@ -0,0 +1,154 @@
+# DOC-IR × spider-rs Convergence — Integration Plan v1 (2026-07-13)
+
+> **Status:** PLAN — companion to `OGAR-DOC-LAYER-PROPOSAL.md` (see its
+> 2026-07-13 AMENDMENT A1–A5, which this plan sequences). The doc-layer
+> 5+3 council gates W1+; W0 is verification-only and runs now.
+>
+> **Ask (operator, 2026-07-13):** combine recognition and
+> processing/representation of knowledge **whether it comes from
+> tesseract-rs or spider-rs** (`AdaWorldAPI/spider`, fork of
+> `spider-rs/spider`) — "using the JSON-like structuring as a reusable
+> input awareness (2D-'spatial' focus of attention)."
+>
+> **One sentence:** the `doc.v1` structure is promoted from "OCR output"
+> to the substrate's **perceptual IR** (`ogar-doc-ir`), with tesseract-rs
+> (pixels) and spider-rs (DOM) as its two sanctioned producers — the
+> code-side pattern (N language frontends → one closed ndjson → one
+> ModelGraph) replayed on perception, so everything upstream (ogar-doc
+> persistence, DeepNSM, lance-graph discovery, MedCare-rs as abstract
+> document store) consumes ONE shape regardless of retina.
+
+## P0. The convergence thesis (why one IR, stated once)
+
+| code side (shipped) | perception side (this plan) |
+|---|---|
+| Roslyn / ruff_python / ruff_ruby frontends | tesseract-rs (scan) / spider-rs (web) retinas |
+| closed 62-predicate ndjson, `from_ndjson` hard-fail gate | closed region-kind doc IR, `from_json` hard-fail gate |
+| `ruff_spo_triplet::ModelGraph` | `ogar-doc-ir` region tree |
+| `CompiledClass` in the graph | `document 0x080B` awareness subtree |
+
+spider-rs is an unusually good second retina because **HTML5 already
+declares its regions**: `<header>/<main>/<footer>/<table>` are the literal
+Kopfzeile/Fußzeile/main positions the amendment's A3 template projects —
+the web *self-labels* what OCR must *infer*. Same regions, two
+acquisition modes, one IR.
+
+## P1. Access preflight (blocking facts to verify, first spider-enabled session)
+
+`AdaWorldAPI/spider` was NOT readable from this session (repo-access gate).
+Before W3 is scoped in detail, a session with spider access verifies:
+
+- **PF-1** fork exists, default branch, divergence from upstream
+  `spider-rs/spider`.
+- **PF-2** page output surface (assumed from upstream: `Website` crawl →
+  per-page HTML string + URL + headers; a headless-chrome feature exists
+  for rendered layout). Verify names/features before writing the seam.
+- **PF-3** whether rendered layout coordinates are obtainable (headless
+  feature) or whether W3 ships DOM-order pseudo-geometry first (see W3
+  fallback — the plan works either way).
+- **PF-4** workspace layout: where a harvest crate lands in the fork
+  (sibling crate, mirroring how `ruff_*_spo` crates live in the ruff fork).
+
+## P2. Wave plan
+
+### W0 — council gate (no code)
+
+The doc-layer 5+3 council verifies the AMENDED proposal: mints
+`typed_field 0x080A` + `document 0x080B`; one-crate `ogar-doc`;
+`ogar-doc-ir` as neutral tissue; the A2 one-leg runtime-binding rule; the
+A3 ClassView × WideFieldMask template. **The council verifies the
+source-agnostic boundary — a spec that only a tesseract producer could
+satisfy is a numbered factual error.**
+
+### W1 — `ogar-doc-ir` (the IR crate; gates everything downstream)
+
+- Types: region tree (page / region / table+cells / figure / header /
+  footer), typed fields, quality.
+- **Closed region-kind vocabulary**; `from_json` HARD-FAILS on unknown
+  kinds; explicit `doc.v1` version marker (v2 = a new marker, never a
+  silent reshape).
+- **Spatial rails:** every bbox quantized to u8 × u8 unit square (the
+  `X:Y` rail; the page IS a 256×256 tile on the existing centroid/Morton
+  machinery). Raw coordinates = optional provenance, never the address.
+- **Provenance lane:** source enum (`ocr` / `dom`) + per-source confidence
+  semantics (recognition confidence ≠ crawl trust; no shared float).
+- **Identity:** content sha256 as the subtree dedup key.
+- Deps: serde only. Tests: vocab closure, gate hard-fail on unknown kind,
+  rail quantization round-trip, version-marker refusal.
+
+### W2 — tesseract-rs producer migration (byte-parity gated)
+
+`structured::{DocPage, render_json_with_regions, …}` emits `ogar-doc-ir`
+types; the **existing doc.v1 JSON output is the committed golden** and the
+migration must reproduce it byte-for-byte (same discipline as the sono
+parity witness: the shipped artifact gates its own refactor). tesseract-rs
+gains a dep on `ogar-doc-ir`; `ogar-doc-ir` never depends back.
+
+### W3 — spider-fork producer (`spider_doc_ir` harvest crate, in the fork)
+
+DOM → doc IR: HTML5 landmarks → region kinds; DOM order → reading order;
+`<table>` → cell grids; microdata/schema.org → typed fields; rendered
+layout coords → rails when PF-3 confirms the headless path, else
+**DOM-order pseudo-geometry** (top-to-bottom rail assignment) as the
+honest v1 — provenance lane marks which geometry mode produced the rails.
+Fixture-driven tests on committed HTML files; no live crawling in CI.
+Lives in the spider fork (PF-4), mirroring `ruff_*_spo`-in-ruff; depends
+on `ogar-doc-ir` via git dep floating on main (D-NEVER-PIN-BUMP).
+
+### W4 — `ogar-doc` (persistence + template + renderer trait)
+
+Per the amended proposal: `persist_document` / `read_document` /
+`reconstruct_document` facts; `DocTemplate` = ClassView × WideFieldMask
+(A3); `trait DocRenderer` with tesseract / Spire.Doc / askama adapters
+bound at runtime (A2). First consumer: MedCare-rs `routes/dms.rs` upload
+→ `recognize_document` → `persist_document` (raw-ref = encrypted
+`file_filelist` key) — sonography, histology, invoices as the abstract
+document store.
+
+### W5 — upstream foresight seams (small, deliberate, no redesign)
+
+- **DeepNSM** (stays 0-dep): entry point accepts
+  `(text_span, region_kind, spatial_rail)` tuples in reading order — the
+  tuple IS the contract; no doc-schema dep enters the crate. Gives the
+  FSM region priors (a table cell is not a sentence), the temporal stream
+  its reading order (per the temporal-stream ruling), and spatial
+  adjacency (caption-near-figure).
+- **lance-graph discovery arm:** `document 0x080B` registered as a
+  discoverable concept; sha256 as the convergence key. Typed fields are
+  already SPO facts + rails — "all invoices from X with IBAN Y" is a
+  graph query + rail scan, no new machinery.
+
+## P3. Probes (falsifiable; graded [H] until run)
+
+- **P-XRETINA (the killer witness):** the same invoice through both
+  retinas (scan → tesseract; HTML → spider) must converge to the same
+  typed-field facts and the same sha-keyed subtree identity. Diverse
+  redundancy applied to perception — the MySQL-oracle pattern. GREEN
+  promotes the whole IR from [S] to [G]; RED means the IR is
+  tesseract-shaped-with-a-second-door and W1 gets corrected before more
+  producers land.
+- **P-DOM-GROUNDTRUTH:** spider + headless render yields (pixels,
+  DOM-labeled regions) pairs at zero annotation cost — a calibration
+  corpus for tesseract's region classifier. The web trains the eye that
+  reads paper.
+- **P-2D-MARKOV:** does spatial-neighborhood attention (Morton rail
+  adjacency) beat pure reading-order for field harvest? Slots into the
+  existing temporal-stream probe queue; not a W-gate.
+
+## P4. Non-goals (v1 of this plan)
+
+- No live crawling in any CI path (fixtures only).
+- No spider-side scheduling/politeness/auth policy — the fork's own
+  concern; this plan touches only the page → doc-IR seam.
+- No multi-language OCR slot, no streaming payloads, no storage-backend
+  choice (all inherited deferrals from the doc-layer proposal).
+- No DeepNSM redesign — W5's tuple signature is the entire ask.
+
+## P5. Sequencing + gates
+
+`W0 (council) → W1 (IR) → {W2, W3} in parallel → P-XRETINA → W4 → W5`.
+W2/W3 are independent once W1 lands; **P-XRETINA runs the moment both
+producers exist and BEFORE W4 persists anything** — persisting a
+tesseract-shaped subtree and discovering the divergence later would bake
+the bias into stored data. Merge-order discipline per house rule:
+OGAR-side crates first, fork-side producers second, in lockstep.

docs/OGAR-DOC-LAYER-PROPOSAL.md

@@ -145,3 +145,122 @@ Append `D-OGAR-DOC-LAYER` to `docs/DISCOVERY-MAP.md`: `ogar-doc` persists `doc.v
 as a raw-ref (KV) + an awareness subtree (SoA graph), reconstructs it via a
 `reconstruct_document` ActionDef; triggers the v2-deferred `typed_field` mint +
 a `document` mint. Grade `[S]` (spec) until the council + a probe promote it.
+
+---
+
+# AMENDMENT — operator rulings, 2026-07-13 (pre-council; the council verifies THIS boundary)
+
+> Supersedes the body above where they conflict. Four rulings from the
+> 2026-07-13 session, each recorded with its reasoning so the 5+3 council
+> verifies rather than re-litigates. Companion integration plan (the
+> spider-rs convergence + wave sequencing): `docs/DOC-IR-SPIDER-CONVERGENCE-PLAN.md`.
+
+## A1. ONE `ogar-doc` crate; the split axis is STATE, not data direction
+
+The considered-and-rejected topology was `ogar-ocr (inbound)` /
+`ogar-doc (outbound)`. Rejected because "render" names two different things:
+
+- `render_searchable_pdf` / `render_text` / `render_tsv` / `render_hocr`
+  (ocr_actions rows 5–8) are the **OCR round-trip** — serialize *this scan's*
+  recognition back onto the pixels it came from. Stateless tesseract-rs
+  compute; they STAY in the OCR surface (already shipped, unchanged).
+- `reconstruct_document` never sees pixels — it walks **stored graph
+  knowledge** and lays it out. Stateful; it belongs to `ogar-doc` *because it
+  reads the graph*, not because it is "outbound."
+
+So: `ogar-ocr` (the existing ocr_actions surface) = **perception**, stateless,
+pixels ↔ structure ↔ bytes. `ogar-doc` = **memory + lifecycle**
+(`persist_document` / `read_document` / `reconstruct_document` + the template),
+one crate, in and out. Creating Word/PDF documents from graph knowledge is NOT
+a tesseract-rs concern, even though tesseract-rs serves delicate PDF features
+`ogar-doc` composes.
+
+## A2. `DocRenderer` is a TRAIT; tesseract-rs is one adapter (the one-leg rule)
+
+tesseract-rs rendering capabilities become **explicitly reusable** behind a
+renderer trait owned by `ogar-doc`:
+
+```text
+ogar-doc:  DocTemplate (ClassView × WideFieldMask) ──▶ trait DocRenderer
+                                                        ├─ tesseract adapter (raster + text-layer PDF)
+                                                        ├─ Spire.Doc adapter  (Word, C#-side)
+                                                        └─ askama adapter     (data-only PDF/HTML)
+```
+
+**The one boundary the council must hold:** the tesseract adapter binds at
+RUNTIME (ActionInvocation / hot-plug, same fuse pattern as `tesseract-ogar`),
+never as a compile-time `use` — otherwise `ogar-doc` drags the tesseract
+render brick into its compile graph and quietly recreates the second leg the
+topology avoids. One tesseract-rs leg total, at the executor layer, serving
+both recognition and rendering.
+
+## A3. The document template IS ClassView × WideFieldMask (no new DSL)
+
+A document layout — Kopfzeile / main / Fußzeile — is not a new template
+vocabulary. It is the SAME machinery as the Klickwege structure surface:
+
+- **ClassView** projects the document class's ordered region set (header
+  positions, body/table, footer positions) exactly as it projects a class's
+  field set — and §D2 already has the ClassView projecting the 4+12 facet
+  register on the *inbound* side.
+- **WideFieldMask** gates which regions are present in this instance —
+  the same `from_universe_present` brick that gates screens in the
+  Klickwege menu-quad.
+- Header/footer/main are **positions in the ordered set**, structurally
+  identical to screen-surface regions.
+
+Consequence: ClassView projects `doc.v1 → facets` going in and
+`facets → laid-out document` going out; WideFieldMask gates presence both
+ways; navigation surfaces and documents render through ONE mechanism. This
+SHARPENS D8's "reuse ogar-render-askama" — the template contract is
+ClassView × WideFieldMask; askama is merely one `DocRenderer` adapter's
+serialization. Spire.Doc reuses the same positional projection for Word.
+
+## A4. `ogar-doc-ir` — the doc IR is SOURCE-AGNOSTIC neutral tissue (the second-retina rule)
+
+The `doc.v1` structure is not an OCR output format — it is the **perceptual
+IR**, and it is about to gain a second producer: **spider-rs**
+(`AdaWorldAPI/spider` fork) harvesting web pages, where HTML5 sectioning
+(`<header>/<main>/<footer>/<table>`) *self-labels* the same regions OCR must
+infer. This is the code-side pattern replayed on perception: N language
+frontends → one closed ndjson → one ModelGraph becomes N perception
+frontends (retinas) → one closed doc IR → one awareness subtree.
+
+Therefore the IR types move OUT of tesseract-rs ownership into a new
+**`ogar-doc-ir`** crate (near-zero-dep: serde only), with the
+`ruff_spo_triplet` discipline applied verbatim:
+
+1. **Closed region-kind vocabulary** + a `from_json` load gate that
+   HARD-FAILS on unknown kinds; explicit version marker (`doc.v1` → `doc.v2`).
+2. **Canonical spatial address = u8-quantized unit square per axis** — the
+   bbox IS the `X:Y` rail of the 4+12 facet; the page/viewport becomes a
+   256×256 tile on the existing centroid/Morton machinery. Raw pixel/viewport
+   coordinates are optional provenance, never the address. This is the
+   "2D spatial focus of attention": a consumer attends to regions across
+   documents from keys alone, zero value decode.
+3. **Provenance lane**: source enum (`ocr` / `dom` / …) with PER-SOURCE
+   confidence semantics — OCR recognition confidence and crawl/boilerplate
+   trust are different quantities and must never alias in one float
+   (I-LEGACY-API-FEATURE-GATED applied prospectively).
+4. **Subtree identity = content sha256** — the same document arriving as
+   scan and as HTML converges on ONE `document 0x080B` subtree. This is the
+   dedup pin arm-discovery needs.
+
+Producers (tesseract-rs's `structured::render_json*` arm; the spider fork's
+future harvest crate) depend on `ogar-doc-ir` — never the reverse, and never
+on each other. The existing tesseract doc.v1 JSON becomes the byte-parity
+golden for the migration (no silent reshape).
+
+## A5. Net crate topology (what the council ratifies)
+
+| crate | owns | deps |
+|---|---|---|
+| `ogar-doc-ir` | doc IR types, closed vocab, load gate, rails, provenance, sha256 identity | serde only |
+| `ogar-doc` | `persist` / `read` / `reconstruct` ActionDefs' vocab side + `DocTemplate` (ClassView × WideFieldMask) + `trait DocRenderer` | ogar-vocab, ogar-doc-ir |
+| ocr surface (existing) | rows 1–14 incl. render primitives | unchanged |
+| producers | tesseract-rs `structured` arm; spider-fork harvest crate | → ogar-doc-ir |
+| renderer adapters | tesseract PDF / Spire.Doc / askama | runtime-bound (A2) |
+
+Mints unchanged from D3 (`typed_field 0x080A`, `document 0x080B`); the three
+ActionDefs unchanged from D4. D8's "no template DSL" is KEPT and sharpened
+(A3); D8's "no storage backend chosen" is KEPT.