Skip to content

Commit 766c981

Browse files
author
Halleluyah Oladipo
committed
feat(ingest): LLM-driven TOC tree builder (PageIndex-style)
Ports the three-phase PageIndex pipeline into Go: 1. detect — single-page TOC detector over the first N pages. 2. extract — if a TOC page was found, parse it into nested nodes; otherwise fall through to the no-TOC path that generates a TOC straight from body text tagged with <physical_index_X> markers. 3. verify — concurrently re-check every node's claimed start page; mismatches clear the page back to zero (the "unknown / open" sentinel) rather than making one up. End pages are derived from sibling ordering after verification. Node IDs are stamped deterministically from the dotted structure so callers can diff trees across re-ingestions. The retry-on-parse-failure helper mirrors pkg/retrieval/single_pass.go::runSelectionWithRetry; it is duplicated rather than imported so the builder doesn't drag the retrieval package into its dependency graph. LLM parse blips degrade to "no usable nodes" with a logged warning so a single bad response never fails ingest — the document remains fully retrievable via the existing sections tree. Tests cover the happy path, the no-TOC path, verification repair, JSON retry, end-page derivation, hierarchy assembly, the <physical_index_X> tag parser, and the empty-input short-circuit.
1 parent 427682f commit 766c981

2 files changed

Lines changed: 1200 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)