Skip to content

Commit 0cda733

Browse files
committed
fix(ingest): bound the whole PDF parse with a configurable deadline
A 10-K was observed hanging 600s+ in `parsing` even in minimal mode: the hang is in ledongthuc row extraction (extractPDFRows -> reader.Page(n).Content()), which is pure-Go and pre-LLM, so none of the existing per-stage table-extraction budgets bound it. Wrap the entire PDF parse (row extraction, table extraction, section building, leaf cap) in a deadline. The work runs on a goroutine; on timeout or ctx cancellation Parse returns a clear error and abandons the goroutine (buffered result channel, so no leak on send; a panic in the work is recovered). This is the same abandon-on-deadline pattern safeExtractTables already uses for a single table page, lifted to cover the whole parse so ANY parse pathology fails fast and cleanly instead of wedging ingest. The ingest pipeline already treats a parse error as a doc-level failure, so the document goes to `failed` and is visible to ops/bench rather than hanging forever. Nothing is disabled: the full feature set (LLM TOC, tables, summarize, HyDE, multi-axis) still runs — parse is merely bounded. Config: IngestConfig.ParseTimeoutSeconds (default 120). Env VLE_INGEST_PARSE_TIMEOUT_SECONDS; the server binary forwards VLS_/VLE_INGEST_PARSE_TIMEOUT_SECONDS. Also threads the existing ingest.max_sections through to the parser (previously dropped on the floor) via RegistryFromIngestParams, so both robustness valves are operator-tunable. A negative value disables the bound (escape hatch). Tests: the deadline mechanism returns a timeout error in ~the timeout (not after a 10s sleep), passes fast work through, propagates real parse errors, runs inline when disabled, honours ctx cancel, and recovers panics; plus config default/env-override/validate coverage.
1 parent 084abc8 commit 0cda733

9 files changed

Lines changed: 474 additions & 11 deletions

File tree

cmd/engine/main.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ func run() error {
172172
DB: pool,
173173
Storage: store,
174174
LLM: llmClient,
175-
Parsers: ingest.RegistryFromTableOpts(tableOptsFromConfig(cfg.Ingest.Tables)),
175+
Parsers: ingest.RegistryFromIngestParams(tableOptsFromConfig(cfg.Ingest.Tables), cfg.Ingest.MaxSections, time.Duration(cfg.Ingest.ParseTimeoutSeconds)*time.Second),
176176
Logger: logger,
177177
Mode: cfg.Ingest.Mode,
178178
HyDEEnabled: cfg.Ingest.HyDE.Enabled,

cmd/server/main.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,7 @@ func run() error {
198198
DB: pool,
199199
Storage: store,
200200
LLM: llmClient,
201-
Parsers: ingest.RegistryFromTableOpts(tableOptsFromConfig(cfg.Engine.Ingest.Tables)),
201+
Parsers: ingest.RegistryFromIngestParams(tableOptsFromConfig(cfg.Engine.Ingest.Tables), cfg.Engine.Ingest.MaxSections, time.Duration(cfg.Engine.Ingest.ParseTimeoutSeconds)*time.Second),
202202
Logger: logger,
203203
Mode: cfg.Engine.Ingest.Mode,
204204
HyDEEnabled: cfg.Engine.Ingest.HyDE.Enabled,

config.example.yaml

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,39 @@ ingest:
320320
# vectorless-server use VLS_INGEST_MODE=minimal (no secret edit needed).
321321
mode: "full"
322322

323+
# Total-parse timeout (seconds). Bounds the ENTIRE parse of one
324+
# document end to end — row extraction, table extraction, section
325+
# building, and the leaf-section cap. It is the outermost robustness
326+
# valve: a pathological/malformed PDF (observed: a 10-K stuck 600s+ in
327+
# `parsing`, even in minimal mode, inside pure-Go row extraction) is
328+
# abandoned at the deadline and the document fails fast instead of
329+
# wedging the pipeline forever. NOTHING is disabled by this bound — the
330+
# full feature set (LLM TOC, tables, summarize, HyDE, multi-axis) still
331+
# runs; parse is merely time-boxed. Applies in BOTH full and minimal
332+
# mode (parse runs in both).
333+
#
334+
# 120 is comfortably longer than a healthy 300-page filing's parse
335+
# (seconds to low tens of seconds) yet short enough to reap a hang
336+
# quickly. 0 uses the engine default (120). Override per-process with
337+
# VLE_INGEST_PARSE_TIMEOUT_SECONDS; the deployed server also honours
338+
# VLS_INGEST_PARSE_TIMEOUT_SECONDS.
339+
parse_timeout_seconds: 120
340+
341+
# Cap on the number of leaf sections one document may produce. A
342+
# pathological PDF (e.g. a 90-page 10-K whose every bold statement
343+
# title trips the heading detector, or a heading→one-body-leaf chain
344+
# repeated hundreds of times) can shatter into far more leaves than the
345+
# document has real sections — each leaf then costs a summarize + HyDE
346+
# + multi-axis LLM call, which is what throttles/stalls full ingest.
347+
# When the parsed leaf count exceeds this cap the parser merges
348+
# sections (smallest first; single-leaf parents collapse into their
349+
# parent) until the document is back under the cap, preserving content
350+
# and never merging table sections. 400 sits comfortably above a real
351+
# filing's section count while still catching the runaway case. 0 uses
352+
# the engine default (400); a negative value disables the cap. Override
353+
# with VLE_INGEST_MAX_SECTIONS.
354+
max_sections: 400
355+
323356
# The summarize and HyDE stages run concurrently. This caps the total
324357
# number of LLM calls in flight across both stages combined, so the
325358
# provider's per-tenant concurrency limit isn't exceeded. 0 disables

internal/config/config.go

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -205,7 +205,7 @@ func Default() Config {
205205
MaxAge: 86400,
206206
},
207207
Governance: GovernanceConfig{
208-
MaxBodySizeBytes: 33554432, // 32 MiB
208+
MaxBodySizeBytes: 33554432, // 32 MiB
209209
DefaultTimeout: 30 * time.Second,
210210
QueryTimeout: 120 * time.Second,
211211
},
@@ -326,6 +326,15 @@ func applyEnvOverrides(c *Config) {
326326
if v := firstEnv("VLS_INGEST_MODE", "VLE_INGEST_MODE"); v != "" {
327327
c.Engine.Ingest.Mode = v
328328
}
329+
// Total-parse timeout (seconds). Forwarded so the deployed server
330+
// honours a tuned parse deadline without a secret/config edit — the
331+
// outermost robustness valve against a parse that hangs (pre-LLM,
332+
// pure-Go row extraction). VLS_-prefixed wins over VLE_.
333+
if v := firstEnv("VLS_INGEST_PARSE_TIMEOUT_SECONDS", "VLE_INGEST_PARSE_TIMEOUT_SECONDS"); v != "" {
334+
if n, err := strconv.Atoi(v); err == nil && n >= 0 {
335+
c.Engine.Ingest.ParseTimeoutSeconds = n
336+
}
337+
}
329338
// Anthropic-compatible gateway overrides (e.g. GLM/Zhipu via
330339
// https://api.z.ai/api/anthropic): base URL + model, so the
331340
// anthropic driver can run a non-Anthropic model without a secret

pkg/config/config.go

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,28 @@ type IngestConfig struct {
125125
// section count (~170-510 with tables) while still catching the
126126
// runaway case. A negative value is rejected by Validate.
127127
MaxSections int `yaml:"max_sections"`
128+
129+
// ParseTimeoutSeconds bounds the ENTIRE parse of a single document —
130+
// row extraction, table extraction, section building, and the leaf
131+
// cap, end to end. It is the outermost robustness valve: every
132+
// per-stage timeout inside the parser (per-page / doc-wide table
133+
// budgets) is bounded by something pre-LLM, but pure-Go row extraction
134+
// (ledongthuc's reader.Page(n).Content()) had no bound, so a
135+
// pathological PDF (observed: a 10-K stuck 600s+ in `parsing` even in
136+
// minimal mode) could hang the parse forever.
137+
//
138+
// When the whole parse exceeds this deadline the parser abandons the
139+
// work and returns a clear error; the ingest pipeline treats it like
140+
// any other parse failure (the document goes to `failed`), so a doc
141+
// that can't parse in time fails fast and is visible to ops/bench
142+
// rather than wedging the pipeline. NOTHING is disabled — the full
143+
// feature set (LLM TOC, tables, summarize, HyDE, multi-axis) still
144+
// runs; parse is merely bounded.
145+
//
146+
// 0 (or omitted) defaults to 120. A negative value is rejected by
147+
// Validate. Engine env override: VLE_INGEST_PARSE_TIMEOUT_SECONDS;
148+
// the server binary also forwards VLS_/VLE_INGEST_PARSE_TIMEOUT_SECONDS.
149+
ParseTimeoutSeconds int `yaml:"parse_timeout_seconds"`
128150
}
129151

130152
// TOCBlock configures the LLM-driven table-of-contents tree
@@ -728,6 +750,7 @@ func Default() Config {
728750
GlobalLLMConcurrency: 12,
729751
LLMCallTimeoutSeconds: 90,
730752
MaxSections: 400,
753+
ParseTimeoutSeconds: 120,
731754
HyDE: HyDEConfig{
732755
Enabled: true,
733756
NumQuestions: 5,
@@ -911,6 +934,11 @@ func applyEnvOverrides(c *Config) {
911934
c.Ingest.MaxSections = n
912935
}
913936
}
937+
if v := os.Getenv("VLE_INGEST_PARSE_TIMEOUT_SECONDS"); v != "" {
938+
if n, err := strconv.Atoi(v); err == nil && n >= 0 {
939+
c.Ingest.ParseTimeoutSeconds = n
940+
}
941+
}
914942
// pdftable-driven table extraction.
915943
if v := os.Getenv("VLE_INGEST_TABLES_ENABLED"); v != "" {
916944
switch strings.ToLower(strings.TrimSpace(v)) {
@@ -1200,6 +1228,9 @@ func (c Config) Validate() error {
12001228
if c.Ingest.MaxSections < 0 {
12011229
return fmt.Errorf("ingest.max_sections must be >= 0, got %d", c.Ingest.MaxSections)
12021230
}
1231+
if c.Ingest.ParseTimeoutSeconds < 0 {
1232+
return fmt.Errorf("ingest.parse_timeout_seconds must be >= 0, got %d", c.Ingest.ParseTimeoutSeconds)
1233+
}
12031234

12041235
switch c.Ingest.Tables.VerticalStrategy {
12051236
case "", "lines", "lines_strict", "text", "explicit":

pkg/config/config_test.go

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,38 @@ func TestDefaultValues(t *testing.T) {
8282
if cfg.Ingest.TOC.TOCCheckPages != 20 {
8383
t.Errorf("ingest.toc.toc_check_pages = %d, want 20", cfg.Ingest.TOC.TOCCheckPages)
8484
}
85+
if cfg.Ingest.ParseTimeoutSeconds != 120 {
86+
t.Errorf("ingest.parse_timeout_seconds = %d, want 120", cfg.Ingest.ParseTimeoutSeconds)
87+
}
88+
if cfg.Ingest.MaxSections != 400 {
89+
t.Errorf("ingest.max_sections = %d, want 400", cfg.Ingest.MaxSections)
90+
}
91+
}
92+
93+
// TestIngestParseTimeoutEnvOverride covers VLE_INGEST_PARSE_TIMEOUT_SECONDS
94+
// — the operator knob that lets a tuned whole-parse deadline reach the
95+
// parser without a config-file edit.
96+
func TestIngestParseTimeoutEnvOverride(t *testing.T) {
97+
prev := os.Getenv("VLE_INGEST_PARSE_TIMEOUT_SECONDS")
98+
defer os.Setenv("VLE_INGEST_PARSE_TIMEOUT_SECONDS", prev)
99+
100+
os.Setenv("VLE_INGEST_PARSE_TIMEOUT_SECONDS", "300")
101+
cfg := Default()
102+
applyEnvOverrides(&cfg)
103+
if cfg.Ingest.ParseTimeoutSeconds != 300 {
104+
t.Errorf("ingest.parse_timeout_seconds = %d, want 300", cfg.Ingest.ParseTimeoutSeconds)
105+
}
106+
}
107+
108+
// TestIngestParseTimeoutValidate rejects a negative parse timeout — a
109+
// non-positive deadline would silently disable the bound, which must be
110+
// an explicit choice, not a typo that slips through Load.
111+
func TestIngestParseTimeoutValidate(t *testing.T) {
112+
cfg := Default()
113+
cfg.Ingest.ParseTimeoutSeconds = -1
114+
if err := cfg.Validate(); err == nil {
115+
t.Error("Validate should reject a negative ingest.parse_timeout_seconds")
116+
}
85117
}
86118

87119
// TestIngestModeDefault locks the default ingest mode to "full" so the

pkg/ingest/ingest.go

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1122,15 +1122,35 @@ func DefaultRegistry() *parser.Registry {
11221122
}
11231123

11241124
// RegistryFromTableOpts returns a parser.Registry where the PDF parser
1125-
// is configured from the supplied TableOpts. Pass nil to disable table
1126-
// extraction entirely; pass parser.DefaultTableOpts() (or a custom set)
1127-
// to enable. All non-PDF parsers are constructed at their defaults.
1125+
// is configured from the supplied TableOpts, with the leaf-section cap
1126+
// and total-parse timeout left at their parser defaults (400 sections,
1127+
// 120s). Pass nil to disable table extraction entirely; pass
1128+
// parser.DefaultTableOpts() (or a custom set) to enable. All non-PDF
1129+
// parsers are constructed at their defaults.
1130+
//
1131+
// Use RegistryFromIngestParams to thread an operator-tuned cap / parse
1132+
// timeout from config.
11281133
func RegistryFromTableOpts(opts *parser.TableOpts) *parser.Registry {
1134+
return RegistryFromIngestParams(opts, 0, 0)
1135+
}
1136+
1137+
// RegistryFromIngestParams returns a parser.Registry where the PDF parser
1138+
// is configured from the supplied TableOpts AND the operator-tuned
1139+
// leaf-section cap and total-parse timeout (ingest.max_sections,
1140+
// ingest.parse_timeout_seconds). maxSections == 0 / parseTimeout == 0
1141+
// each select the parser's built-in default; a negative value disables
1142+
// that bound. All non-PDF parsers are constructed at their defaults.
1143+
//
1144+
// This is the constructor the engine/server wiring uses so the parse
1145+
// deadline and section cap from config actually reach the parser — the
1146+
// outermost robustness valves for full-feature ingest. Table extraction,
1147+
// the section tree, and the cap all still run; they are merely bounded.
1148+
func RegistryFromIngestParams(opts *parser.TableOpts, maxSections int, parseTimeout time.Duration) *parser.Registry {
11291149
return parser.NewRegistry(
11301150
parser.NewMarkdown(),
11311151
parser.NewHTML(),
11321152
parser.NewDOCX(),
1133-
parser.NewPDFWithTables(opts),
1153+
parser.NewPDFWithConfig(opts, maxSections, parseTimeout),
11341154
parser.NewText(),
11351155
)
11361156
}

pkg/parser/pdf.go

Lines changed: 133 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,30 @@ type PDF struct {
6464
// Zero selects defaultMaxLeafSections. A negative value disables the
6565
// cap entirely (escape hatch for callers that want the raw outline).
6666
MaxSections int
67+
68+
// ParseTimeout bounds the ENTIRE Parse of one document — row
69+
// extraction, table extraction, section building, and the leaf cap,
70+
// end to end. It is the outermost robustness valve.
71+
//
72+
// Every other time bound inside the parser caps a sub-stage
73+
// (tableExtractPageTimeout / tableExtractDocBudget cap table
74+
// extraction), but the pure-Go row extractor
75+
// (extractPDFRows -> reader.Page(n).Content()) had no bound at all —
76+
// and that is exactly where a pathological PDF was observed hanging
77+
// 600s+ in `parsing`, even in minimal mode (pre-LLM). ParseTimeout
78+
// runs the whole parse on a goroutine and abandons it on deadline,
79+
// returning a clear error so ingest fails the document fast instead
80+
// of wedging forever.
81+
//
82+
// Nothing is disabled by this bound: when the parse finishes inside
83+
// the deadline the full feature set (tables, the outline/heuristic
84+
// section tree, the cap) is produced exactly as before.
85+
//
86+
// Zero selects defaultParseTimeout (120s). A non-positive value other
87+
// than zero (i.e. negative) disables the bound entirely — Parse then
88+
// runs synchronously with no deadline (escape hatch / legacy
89+
// behaviour for callers that want an unbounded parse).
90+
ParseTimeout time.Duration
6791
}
6892

6993
// TableOpts controls pdftable's table-finding stage. The zero value
@@ -119,13 +143,24 @@ func NewPDFWithTables(opts *TableOpts) *PDF { return &PDF{Tables: opts} }
119143

120144
// NewPDFWithOpts returns a PDF parser using the supplied table-extraction
121145
// options and an explicit leaf-section cap. maxSections == 0 selects
122-
// defaultMaxLeafSections; a negative value disables the cap. This is the
123-
// constructor the engine wiring uses so the cap is operator-tunable via
124-
// config (ingest.max_sections).
146+
// defaultMaxLeafSections; a negative value disables the cap. The parse
147+
// timeout is left at its zero value, which resolves to defaultParseTimeout.
125148
func NewPDFWithOpts(opts *TableOpts, maxSections int) *PDF {
126149
return &PDF{Tables: opts, MaxSections: maxSections}
127150
}
128151

152+
// NewPDFWithConfig returns a PDF parser with the table options, leaf-
153+
// section cap, AND total-parse timeout all set explicitly. This is the
154+
// constructor the engine wiring uses so every parser robustness knob is
155+
// operator-tunable via config (ingest.max_sections,
156+
// ingest.parse_timeout_seconds).
157+
//
158+
// - maxSections == 0 → defaultMaxLeafSections; negative disables the cap.
159+
// - parseTimeout == 0 → defaultParseTimeout; negative disables the bound.
160+
func NewPDFWithConfig(opts *TableOpts, maxSections int, parseTimeout time.Duration) *PDF {
161+
return &PDF{Tables: opts, MaxSections: maxSections, ParseTimeout: parseTimeout}
162+
}
163+
129164
// Name implements Parser.
130165
func (*PDF) Name() string { return "pdf" }
131166

@@ -138,11 +173,87 @@ func (*PDF) Accepts(contentType, filename string) bool {
138173
}
139174

140175
// Parse implements Parser.
141-
func (p *PDF) Parse(_ context.Context, r io.Reader) (*ParsedDoc, error) {
176+
//
177+
// Parse is a thin timeout wrapper around the real parse work (parseDoc).
178+
// It bounds the WHOLE parse — row extraction, table extraction, section
179+
// building, and the leaf cap — by p.resolvedParseTimeout(). The work runs
180+
// on a goroutine; if it doesn't finish by the deadline (or ctx is
181+
// cancelled) Parse returns a clear error and abandons the goroutine. The
182+
// goroutine then finishes on its own (or is GC'd) — its result lands in a
183+
// buffered channel so it never blocks on send. This is the same
184+
// abandon-on-deadline pattern safeExtractTables already uses for a single
185+
// table page, lifted to cover the entire parse so ANY parse pathology
186+
// (notably the unbounded pure-Go ledongthuc row extractor) fails fast and
187+
// cleanly instead of hanging ingest forever.
188+
//
189+
// A negative ParseTimeout disables the bound and runs parseDoc inline.
190+
func (p *PDF) Parse(ctx context.Context, r io.Reader) (*ParsedDoc, error) {
191+
// Read the bytes up front (outside the deadline goroutine): io.ReadAll
192+
// is bounded by the reader/storage layer, not by the PDF pathology we
193+
// are guarding against, and reading here keeps the goroutine body a
194+
// pure CPU/parse unit.
142195
buf, err := io.ReadAll(r)
143196
if err != nil {
144197
return nil, err
145198
}
199+
return runParseWithDeadline(ctx, p.resolvedParseTimeout(), func() (*ParsedDoc, error) {
200+
return p.parseDoc(ctx, buf)
201+
})
202+
}
203+
204+
// runParseWithDeadline runs work bounded by timeout. It is the reusable
205+
// core of the whole-Parse robustness valve, factored out so the
206+
// deadline/abandon mechanism is unit-testable with an arbitrary work
207+
// closure (e.g. one that sleeps past the deadline) without needing a
208+
// pathological real PDF.
209+
//
210+
// - timeout <= 0 runs work inline with NO bound (legacy/unbounded
211+
// behaviour; the explicit escape hatch a negative ParseTimeout selects).
212+
// - Otherwise work runs on a goroutine and the function selects on the
213+
// work result, a time.After(timeout), and ctx cancellation. On
214+
// timeout/cancel the goroutine is abandoned: its result lands in a
215+
// buffered channel so it can always send and exit (no leak on send),
216+
// then runs to completion on its own and is collected. A panic inside
217+
// work is recovered and surfaced as an error so a backend bug can
218+
// never crash the ingest worker.
219+
//
220+
// The timeout/cancel errors are deliberately phrased so the ingest
221+
// pipeline's existing "parse failed → document failed" path produces a
222+
// clear, ops-visible message instead of an infinite hang.
223+
func runParseWithDeadline(ctx context.Context, timeout time.Duration, work func() (*ParsedDoc, error)) (*ParsedDoc, error) {
224+
if timeout <= 0 {
225+
return work()
226+
}
227+
228+
type result struct {
229+
doc *ParsedDoc
230+
err error
231+
}
232+
done := make(chan result, 1)
233+
go func() {
234+
defer func() {
235+
if rec := recover(); rec != nil {
236+
done <- result{err: fmt.Errorf("pdf: parse panicked: %v", rec)}
237+
}
238+
}()
239+
doc, perr := work()
240+
done <- result{doc: doc, err: perr}
241+
}()
242+
243+
select {
244+
case res := <-done:
245+
return res.doc, res.err
246+
case <-time.After(timeout):
247+
return nil, fmt.Errorf("pdf: parse exceeded %s — document too complex or malformed", timeout)
248+
case <-ctx.Done():
249+
return nil, fmt.Errorf("pdf: parse cancelled: %w", ctx.Err())
250+
}
251+
}
252+
253+
// parseDoc is the real parse implementation. It is bounded by Parse's
254+
// deadline wrapper; on its own it has no time bound beyond the per-stage
255+
// table-extraction budgets.
256+
func (p *PDF) parseDoc(_ context.Context, buf []byte) (*ParsedDoc, error) {
146257

147258
// We run TWO PDF backends in parallel here:
148259
//
@@ -407,6 +518,24 @@ func (p *PDF) resolvedMaxSections() int {
407518
return p.MaxSections
408519
}
409520

521+
// defaultParseTimeout is the whole-Parse deadline applied when
522+
// ParseTimeout is left at its zero value. 120s is comfortably longer than
523+
// a healthy 300-page filing's parse (seconds to low tens of seconds) yet
524+
// short enough that a pathological/malformed document — the kind observed
525+
// hanging 600s+ in pure-Go row extraction — is reaped quickly and the
526+
// document fails fast instead of wedging ingest.
527+
const defaultParseTimeout = 120 * time.Second
528+
529+
// resolvedParseTimeout turns the configured ParseTimeout into the value
530+
// the wrapper actually uses: 0 selects defaultParseTimeout; a negative
531+
// value disables the bound (Parse runs parseDoc inline with no deadline).
532+
func (p *PDF) resolvedParseTimeout() time.Duration {
533+
if p.ParseTimeout == 0 {
534+
return defaultParseTimeout
535+
}
536+
return p.ParseTimeout
537+
}
538+
410539
// propagateSectionPages fills internal-node PageStart/PageEnd from the union
411540
// of descendant leaf ranges where the internal node didn't have its own
412541
// (because its body was empty / hoisted into children). Leaves keep their

0 commit comments

Comments
 (0)