Commit 766c981
Halleluyah Oladipo
feat(ingest): LLM-driven TOC tree builder (PageIndex-style)
Ports the three-phase PageIndex pipeline into Go:
1. detect — single-page TOC detector over the first N pages.
2. extract — if a TOC page was found, parse it into nested
nodes; otherwise fall through to the no-TOC
path that generates a TOC straight from body
text tagged with <physical_index_X> markers.
3. verify — concurrently re-check every node's claimed
start page; mismatches clear the page back to
zero (the "unknown / open" sentinel) rather
than making one up.
End pages are derived from sibling ordering after verification.
Node IDs are stamped deterministically from the dotted
structure so callers can diff trees across re-ingestions.
The retry-on-parse-failure helper mirrors
pkg/retrieval/single_pass.go::runSelectionWithRetry; it is
duplicated rather than imported so the builder doesn't drag the
retrieval package into its dependency graph.
LLM parse blips degrade to "no usable nodes" with a logged
warning so a single bad response never fails ingest — the
document remains fully retrievable via the existing sections
tree.
Tests cover the happy path, the no-TOC path, verification
repair, JSON retry, end-page derivation, hierarchy assembly,
the <physical_index_X> tag parser, and the empty-input
short-circuit.1 parent 427682f commit 766c981
2 files changed
Lines changed: 1200 additions & 0 deletions
0 commit comments