Redesign the extraction contract for grouped, parallel LLM calls
Background
Once the incident contract grew to cover a full incident report, it became too large to hand to the LLM in one shot and expect a clean result. This came up directly in discussion #609, structured extraction quality drops, so a single big pass is not viable for a schema this size.
The extraction endpoints also still assumed the old "extraction produces a standalone blob" model, with no link to a stored incident, and no way for the UI to tell which forms are ready to generate before the user tries and gets a 422.
What needs to change
- Describe the extraction as multiple small grouped calls run in parallel against Ollama (bounded by
OLLAMA_NUM_PARALLEL), each validated against its slice of the schema with retry, then stitched into one document. This replaces the single-pass description.
- Extraction now creates a draft incident row the moment it completes. That row holds the contract document and becomes the single store of incident data. The extraction record itself keeps only job metadata and the corrections audit trail, nothing duplicated.
- Add a
defaults block to the extraction request (country, timezone, currency) so relative dates and money fields resolve correctly without the LLM guessing.
- Add
GET /extract/{extract_id}/readiness, a pure lookup endpoint that compares the extracted contract against every registered template's fields and reports which templates are ready to generate and what is blocking the rest. This is what drives the form-selection screen instead of the user finding out via a failed generate call.
- Add
FieldGap to describe a missing field with enough context for the UI to explain it (which template field, where the value should come from, whether it needs a manual entry or a contract correction).
- Switch
validate from a form_type enum to a template_id, since templates are user-registered and form_type is no longer a closed set.
- Support "open" template fields, fields the incident contract does not cover, extracted as extra grouped calls in the same run and stored under the contract's
custom_fields.
Redesign the extraction contract for grouped, parallel LLM calls
Background
Once the incident contract grew to cover a full incident report, it became too large to hand to the LLM in one shot and expect a clean result. This came up directly in discussion #609, structured extraction quality drops, so a single big pass is not viable for a schema this size.
The extraction endpoints also still assumed the old "extraction produces a standalone blob" model, with no link to a stored incident, and no way for the UI to tell which forms are ready to generate before the user tries and gets a 422.
What needs to change
OLLAMA_NUM_PARALLEL), each validated against its slice of the schema with retry, then stitched into one document. This replaces the single-pass description.defaultsblock to the extraction request (country, timezone, currency) so relative dates and money fields resolve correctly without the LLM guessing.GET /extract/{extract_id}/readiness, a pure lookup endpoint that compares the extracted contract against every registered template's fields and reports which templates are ready to generate and what is blocking the rest. This is what drives the form-selection screen instead of the user finding out via a failed generate call.FieldGapto describe a missing field with enough context for the UI to explain it (which template field, where the value should come from, whether it needs a manual entry or a contract correction).validatefrom aform_typeenum to atemplate_id, since templates are user-registered and form_type is no longer a closed set.custom_fields.