Skip to content

Commit 86dd2c1

Browse files
hallelx2hallelx2
committed
docs: add Document Profiles design + roadmap (domain specialization)
Captures the plan for domain-aware structuring before building it: profiles are the layer that turns the generic heading tree into a typed, navigable map per domain (research papers, medical guidelines). - docs/PROFILES.md — design ("why/what"): the Profile interface (Detect/Apply), Section.Kind + typed metadata + cross-refs, rules-first + one-batched-LLM-call classification, the domain catalog (generic baseline, research-paper, medical-guideline), how it plugs into ingest/retrieval/API, and why-not-RAPTOR/embeddings. Includes a written snapshot of the current structuring MVP as the baseline the design builds on. - docs/roadmaps/PROFILES.md — phased checklist (Phase 1 scaffold + research-paper, Phase 2 metadata + medical-guideline, Phase 3 cross-refs + semantic query). - Wired both into the docs + roadmaps index tables. No engine code changes — design only, per "let's document this first."
1 parent 972386e commit 86dd2c1

4 files changed

Lines changed: 363 additions & 0 deletions

File tree

‎docs/PROFILES.md‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
# Document Profiles
2+
3+
> Domain-aware structuring. The layer that turns a generic heading
4+
> tree into a navigable, *intelligent* map an agent can reason over
5+
> in a specialized domain.
6+
7+
## Purpose
8+
9+
The engine's promise is: hand an agent a navigable map of a document
10+
and it does the rest — no embeddings, no RAG, no vector metrics (see
11+
[ENGINE.md](./ENGINE.md)). Today that map is **structurally generic**:
12+
it knows "this is a heading at depth 2", not "this is the *Methods*
13+
section" or "this is a Grade-A *recommendation*".
14+
15+
Generic structure is a great MVP and the right foundation. But
16+
Vectorless earns its keep in **specialized domains** — research
17+
papers, clinical guidelines, legal contracts, financial filings —
18+
where documents follow strong conventions and an agent benefits from
19+
*semantic* navigation, not just structural. "Jump to the Methods
20+
section." "Show me every recommendation with evidence grade A." "What
21+
were the contributions and how were they validated?"
22+
23+
A **Document Profile** is the unit that encodes a domain's conventions
24+
and projects them onto the map. Profiles are the *guards* that make
25+
Vectorless reliable for a specialized system: a schema that constrains
26+
and enriches the structure so agents navigate meaning, not just
27+
indentation.
28+
29+
## The core idea
30+
31+
A profile takes the generic tree the parser produced and returns the
32+
same tree with three things added:
33+
34+
1. **A `Kind` on every section** — the canonical section type in that
35+
domain (`Abstract`, `Methods`, `Recommendation`, `Dosage`, …).
36+
2. **Typed metadata per section** — structured fields the domain cares
37+
about (`evidence_grade=A`, `dataset=WMT14`, `population=adults`).
38+
3. **Cross-references** — links the raw outline doesn't capture
39+
(citations, "see §3", recommendation → supporting evidence).
40+
41+
The parser still does the structural work (headings → tree). The
42+
profile does the *semantic* work on top. They compose; neither knows
43+
about the other's internals.
44+
45+
```
46+
bytes ──parser──▶ generic tree ──profile──▶ typed, enriched map
47+
(structure) (semantics)
48+
```
49+
50+
## What a profile is
51+
52+
A profile is a pluggable module, the same pattern as `pkg/parser`.
53+
One profile per domain; a registry routes a document to the best fit.
54+
55+
```go
56+
// pkg/profile
57+
58+
// Kind is a domain-canonical section type. Open string, not a global
59+
// enum — each profile defines its own vocabulary as constants.
60+
type Kind string
61+
62+
// CrossRef is a typed edge the raw outline didn't capture.
63+
type CrossRef struct {
64+
From SectionID
65+
To SectionID // or external (a citation key, a URL)
66+
Rel string // "cites" | "evidence_for" | "see_also" | ...
67+
}
68+
69+
type Profile interface {
70+
// Name is the stable id ("research-paper", "medical-guideline").
71+
Name() string
72+
73+
// Detect returns 0..1 confidence that this profile fits the
74+
// document, given its outline + lightweight metadata. Cheap:
75+
// title/heading heuristics only, no LLM, no content fetch.
76+
Detect(t *tree.Tree, meta DocMeta) float64
77+
78+
// Apply classifies every section into a Kind, extracts typed
79+
// metadata, and returns cross-references. May make a small number
80+
// of LLM calls (ideally one batched call per document). The
81+
// tree is annotated in place; cross-refs are returned.
82+
Apply(ctx context.Context, t *tree.Tree, llm llm.Client) ([]CrossRef, error)
83+
}
84+
```
85+
86+
`DocMeta` carries what's known before content fetch: filename,
87+
content-type, page count, byte size, and the flat list of heading
88+
titles. Enough for `Detect` to decide cheaply.
89+
90+
### Classification: rules first, LLM to finish
91+
92+
Most section titles are unambiguous — "Abstract", "3 Methods",
93+
"References", "Contraindications" classify by keyword/regex with zero
94+
model cost. `Apply` runs the rule pass first, then makes **one batched
95+
LLM call** for the residue: "Here is the outline of a research paper;
96+
assign each of these untyped sections to one of {Method, Result,
97+
…}." One call per document, cheap, and the profile's taxonomy bounds
98+
the output so the model can't invent kinds.
99+
100+
This mirrors the engine's whole philosophy: structure does the heavy
101+
lifting; the LLM is used surgically, not as a hammer.
102+
103+
## The Section gains a Kind
104+
105+
`tree.Section` adds a `Kind` field and the `sections` table adds a
106+
`kind` column (indexed). The existing `metadata map[string]string`
107+
column carries the typed fields — no schema churn there.
108+
109+
```
110+
sections
111+
...
112+
kind TEXT NOT NULL DEFAULT '' -- canonical section type
113+
metadata JSONB -- typed domain fields (exists)
114+
```
115+
116+
The tree `View` (what the agent reasons over) exposes `kind` next to
117+
`title` + `summary`, so the ToC the agent reads is already typed:
118+
119+
```
120+
[Abstract] Attention Is All You Need — abstract
121+
[Method] §3 Model Architecture — encoder/decoder with multi-head attention
122+
[Result] §6 Results — BLEU 28.4 on WMT14 EN-DE, new SOTA
123+
```
124+
125+
A `kind` column also unlocks **structural query filters**: "select
126+
only `Recommendation` sections", "list all `Result` nodes" — without
127+
an LLM call at all.
128+
129+
## Domain catalog
130+
131+
### `generic` (baseline, ships first)
132+
133+
A no-op profile: `Kind = ""` for everything, no extraction, no
134+
cross-refs. This is exactly today's behavior. Every document that no
135+
specialized profile claims falls back to `generic`, so adding profiles
136+
is strictly additive — zero regression risk.
137+
138+
### `research-paper`
139+
140+
| | |
141+
|---|---|
142+
| **Detect** | Has an `Abstract` + `References`; arXiv-style filename or DOI; IMRaD-ish heading set. |
143+
| **Kinds** | `Abstract`, `Contributions`, `RelatedWork`, `Method`, `Experiment`, `Result`, `Limitation`, `Conclusion`, `References`, `Appendix`, `Figure`, `Table` |
144+
| **Metadata** | `datasets`, `metrics`, `figure_refs`, `table_refs`, `equation_refs` |
145+
| **Cross-refs** | citation graph (section → reference keys); claim → experiment that supports it |
146+
| **Agent win** | "main contribution + how validated?" → `Contributions` → `Experiment` → `Result` |
147+
148+
### `medical-guideline`
149+
150+
| | |
151+
|---|---|
152+
| **Detect** | Recommendation/evidence-grade phrasing; GINA/NICE/WHO-style structure; ICD/drug terms. |
153+
| **Kinds** | `Recommendation`, `Evidence`, `Population`, `Intervention`, `Dosage`, `Contraindication`, `Monitoring`, `References` |
154+
| **Metadata** | `evidence_grade` (A/B/C/D), `condition`, `drug`, `population`, `strength_of_recommendation` |
155+
| **Cross-refs** | recommendation → its evidence source; intervention → contraindication |
156+
| **Agent win** | "first-line Tx for X + evidence grade?" → filter `Recommendation` by `condition=X`, return with `dosage` + `evidence_grade` |
157+
158+
Future candidates: `legal-contract` (clauses, definitions, exhibits,
159+
obligations), `financial-filing` (10-K MD&A, risk factors,
160+
statements), `api-docs`, `clinical-note` (SOAP).
161+
162+
## Profile selection
163+
164+
Two ways a document gets a profile; **open design question** which to
165+
default to (see roadmap):
166+
167+
- **Declared** — the caller sets it at upload (`X-Vectorless-Profile:
168+
research-paper`, or an SDK arg). Predictable, explicit, best for a
169+
specialized system that *knows* its corpus is all one kind.
170+
- **Auto-detected** — the engine runs every profile's `Detect`, picks
171+
the highest-confidence one above a threshold, falls back to
172+
`generic`. Zero-config, best for mixed corpora.
173+
174+
The likely answer is **both**: honor a declared profile; otherwise
175+
auto-detect; otherwise `generic`. Detection is cheap (no LLM), so
176+
running it always is fine.
177+
178+
## How it plugs into the engine
179+
180+
Minimal, surgical changes — profiles slot between existing stages.
181+
182+
- **`pkg/profile/`** — new package: `Profile` interface, `Registry`,
183+
`generic`, `research-paper`. Same shape as `pkg/parser`.
184+
- **`pkg/tree`** — `Section.Kind` field; `View` exposes it.
185+
- **`pkg/db`** — `kind` column + migration; reads/writes it.
186+
- **`pkg/ingest`** — after `persistTree`, run the selected profile's
187+
`Apply`, persist `kind` + metadata + cross-refs. New lifecycle is
188+
`parsing → structuring → summarizing → ready` (structuring = profile
189+
apply). Idempotent like every other stage.
190+
- **`pkg/retrieval`** — the chunked-tree / single-pass prompt becomes
191+
profile-aware: it's handed the profile's name + kind glossary so the
192+
model reasons with domain context ("Methods describes *how*, Results
193+
describes *outcomes*"). Query API gains an optional `kinds` filter.
194+
- **server / API** — `X-Vectorless-Profile` on upload; `profile` +
195+
per-section `kind` in document/tree responses; `kinds[]` filter on
196+
`/v1/query`.
197+
198+
The `generic` fallback means none of this changes behavior for
199+
documents that don't opt in.
200+
201+
## Why this and not…
202+
203+
- **…RAPTOR / embeddings / a vector tree?** RAPTOR builds a *synthetic*
204+
tree by embedding chunks, clustering them, and recursively
205+
summarizing clusters — it still needs a vector store and similarity
206+
search. That's the exact dependency Vectorless is positioned
207+
against. Profiles enrich the document's *real* structure; no
208+
embeddings enter the system. (If a document has *no* usable
209+
structure — flat OCR, a wall of text — a future `synthetic` profile
210+
could build a tree via recursive summarization *without* embeddings.
211+
That's the one place RAPTOR's idea is worth borrowing, minus the
212+
vectors.)
213+
- **…a fine-tuned section classifier?** Overkill and brittle across
214+
domains. Rules + one bounded LLM call per doc is cheaper, debuggable,
215+
and adapts to a new domain by writing a profile, not retraining.
216+
- **…just better prompts on the generic tree?** Prompts can't add a
217+
`kind` column you can filter on, or a citation graph, or guarantee a
218+
paper has a Methods section. Profiles make the structure *typed and
219+
queryable*, not just better-described.
220+
221+
## Baseline: how structuring works today (the MVP)
222+
223+
So the design builds on a written starting point:
224+
225+
- **Parsers** (`pkg/parser`): Markdown, HTML, DOCX, Text, PDF. Each
226+
returns a `*tree.Tree` from the document's headings. PDF uses a
227+
font-size heuristic (+ the `/Outlines` bookmark table when present),
228+
with recent hardening: pdfcpu decrypt-with-empty-password for
229+
owner-encrypted PDFs, a tuned word-space threshold (0.20·fontSize),
230+
invalid-UTF-8 / control-char scrubbing before storage + LLM, a
231+
mojibake-title guard, and publisher/license boilerplate filtering.
232+
- **Structure**: heading hierarchy → `Section` tree. Multiple
233+
top-level sections are wrapped in a synthetic root so the whole
234+
document is reachable.
235+
- **Summaries**: one LLM sentence per section (parallel via errgroup),
236+
with a truncated-excerpt fallback when the LLM call fails.
237+
- **Retrieval**: `single-pass` (whole tree in one call) or
238+
`chunked-tree` (slice → parallel select → merge). The map is the
239+
index; no embeddings anywhere.
240+
- **Multi-tenancy**: every section is scoped by `org_id` on the parent
241+
document; reads/writes filter by it.
242+
243+
Profiles are the **next layer up**: same parsers, same retrieval, same
244+
no-embeddings stance — a semantic typing + enrichment pass in between.
245+
246+
## Open questions
247+
248+
- **Default selection** — declared-only, auto-only, or both? (Leaning
249+
both.) What confidence threshold makes auto-detect safe?
250+
- **Kind taxonomy stability** — are kinds free strings per profile, or
251+
is there a shared cross-domain core (`Body`, `Reference`, `Meta`)
252+
every profile maps into for generic tooling?
253+
- **Re-profiling** — if a better profile ships later, do we re-apply
254+
to existing docs? (Tie into the incremental re-ingest work in
255+
[ENGINE.md](./ENGINE.md) open questions.)
256+
- **Cross-refs storage** — new `section_refs` table vs. edges in
257+
section metadata. A table is queryable; metadata is cheaper.
258+
- **Figures/tables as nodes** — first-class `Section`s with a `Figure`
259+
kind, or metadata refs on the owning section? Affects how an agent
260+
addresses "Table 2".
261+
262+
## Related docs
263+
264+
- [ENGINE.md](./ENGINE.md) — the core engine this builds on.
265+
- [DATA.md](./DATA.md) — Postgres vs object storage; the `sections`
266+
schema the `kind` column extends.
267+
- [roadmaps/PROFILES.md](./roadmaps/PROFILES.md) — the phased delivery
268+
plan.

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ specific subsystem you're working on.
2323
| [ARCHITECTURE.md](./ARCHITECTURE.md) | The whole stack, layer by layer. Read this first. |
2424
| [REPOS.md](./REPOS.md) | Which repositories exist, which are public vs private, when to split. |
2525
| [ENGINE.md](./ENGINE.md) | The core retrieval engine — library + daemon. |
26+
| [PROFILES.md](./PROFILES.md) | Domain-aware structuring — typed, navigable maps per domain. |
2627
| [SERVER.md](./SERVER.md) | The HTTP + gRPC service that fronts the engine. |
2728
| [LLMGATE.md](./LLMGATE.md) | The "LiteLLM for Go" gateway layer. |
2829
| [CONTROL-PLANE.md](./CONTROL-PLANE.md) | The SaaS backend — tenants, keys, billing. |

‎docs/roadmaps/PROFILES.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Document Profiles roadmap
2+
3+
> Design doc: [../PROFILES.md](../PROFILES.md)
4+
5+
The *when* and *what's next* for domain-aware structuring. Profiles
6+
turn the generic heading tree into a typed, navigable map for a
7+
specialized domain (research papers, medical guidelines, …).
8+
9+
## Current status (summary)
10+
11+
| Phase | Status |
12+
|---|---|
13+
| Phase 0 — generic baseline (heading tree) | Shipped (the MVP) |
14+
| Phase 1 — profile scaffold + `research-paper` | Not started |
15+
| Phase 2 — domain metadata extraction + `medical-guideline` | Not started |
16+
| Phase 3 — cross-refs + semantic query API | Not started |
17+
18+
The baseline (Phase 0) is what ships today: parsers → heading tree →
19+
per-section summaries → tree retrieval, hardened for real-world PDFs.
20+
Everything below is additive; the `generic` profile preserves current
21+
behavior with zero regression.
22+
23+
## Phase 1 — profile scaffold + research-paper
24+
25+
The foundation. Get one real profile working end-to-end against a
26+
clean test fixture (arXiv "Attention Is All You Need").
27+
28+
- [ ] **`pkg/profile` package**
29+
- [ ] `Profile` interface (`Name`, `Detect`, `Apply`).
30+
- [ ] `Registry` that routes a document to the best-fit profile
31+
(same pattern as `pkg/parser`).
32+
- [ ] `generic` profile — no-op, `Kind=""`. The fallback.
33+
- [ ] `Kind` type + `CrossRef` type + `DocMeta` input.
34+
- [ ] **`tree.Section.Kind`** field; `View` exposes `kind`.
35+
- [ ] **DB**: `kind` column on `sections` (+ migration, indexed).
36+
Reads/writes carry it.
37+
- [ ] **`research-paper` profile**
38+
- [ ] `Detect` from outline (Abstract + References + IMRaD-ish
39+
headings; arXiv/DOI filename hints). No LLM.
40+
- [ ] `Classify` — rule pass (keyword/regex on titles) + one
41+
batched LLM call for the residue, bounded by the kind
42+
vocabulary.
43+
- [ ] Kinds: Abstract, Contributions, RelatedWork, Method,
44+
Experiment, Result, Limitation, Conclusion, References,
45+
Appendix.
46+
- [ ] **Ingest** runs the selected profile after `persistTree`
47+
(`parsing → structuring → summarizing → ready`). Idempotent.
48+
- [ ] **Profile selection**: honor declared `X-Vectorless-Profile`;
49+
else auto-detect; else `generic`. (Resolve the
50+
declared/auto/both open question first.)
51+
- [ ] **Validate** on the arXiv paper: every section gets a sensible
52+
kind; the map's ToC shows `[Method] §3 Model Architecture` etc.
53+
54+
## Phase 2 — domain metadata + medical-guideline
55+
56+
- [ ] **Research metadata extraction**: datasets, metrics, figure /
57+
table / equation refs into section `metadata`.
58+
- [ ] **Figures / tables**: decide nodes vs. metadata refs; implement.
59+
- [ ] **`medical-guideline` profile**
60+
- [ ] `Detect` from recommendation/evidence-grade phrasing.
61+
- [ ] Kinds: Recommendation, Evidence, Population, Intervention,
62+
Dosage, Contraindication, Monitoring, References.
63+
- [ ] Extract `evidence_grade`, `condition`, `drug`, `population`.
64+
- [ ] Validate against a representative guideline PDF (need fixture).
65+
- [ ] **Retrieval becomes profile-aware**: pass the profile name +
66+
kind glossary into the select prompt.
67+
68+
## Phase 3 — cross-refs + semantic navigation
69+
70+
- [ ] **Cross-references**: citation graph for papers; recommendation
71+
→ evidence for guidelines. Decide `section_refs` table vs.
72+
metadata edges.
73+
- [ ] **Query API `kinds[]` filter**: "select only Recommendation
74+
sections" — structural filter, no LLM call.
75+
- [ ] **Claim ↔ evidence links** surfaced in the map + query results.
76+
- [ ] **`synthetic` profile** (optional): build a tree for
77+
structure-less documents (flat OCR) via recursive summarization —
78+
no embeddings.
79+
80+
## Open questions (carry from design doc)
81+
82+
- [ ] Default selection: declared-only / auto-only / both + threshold.
83+
- [ ] Kind taxonomy: free per-profile strings vs. shared cross-domain
84+
core.
85+
- [ ] Re-profiling existing docs when a better profile ships (ties to
86+
incremental re-ingest).
87+
- [ ] Cross-ref storage: table vs. metadata.
88+
89+
## Related
90+
91+
- [../PROFILES.md](../PROFILES.md) — the design doc.
92+
- [../ENGINE.md](../ENGINE.md) — the engine this builds on.
93+
- [../../ROADMAP.md](../../ROADMAP.md) — root checkbox document.

‎docs/roadmaps/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ progress, and what's next.
1616
| Roadmap | Subsystem | Design doc |
1717
|---|---|---|
1818
| [ENGINE.md](./ENGINE.md) | Core engine | [../ENGINE.md](../ENGINE.md) |
19+
| [PROFILES.md](./PROFILES.md) | Domain-aware structuring | [../PROFILES.md](../PROFILES.md) |
1920
| [SERVER.md](./SERVER.md) | HTTP/gRPC transport | [../SERVER.md](../SERVER.md) |
2021
| [LLMGATE.md](./LLMGATE.md) | LLM gateway library | [../LLMGATE.md](../LLMGATE.md) |
2122
| [CONTROL-PLANE.md](./CONTROL-PLANE.md) | SaaS backend | [../CONTROL-PLANE.md](../CONTROL-PLANE.md) |

0 commit comments

Comments
 (0)