fix(parser): harden classic chunker against oversized headers

[FraanW]

What & why

Chunker.split_document (the classic_chunk strategy in application/parser/chunking.py) derives its per-chunk step from max_tokens - len(header_tokens). The header is the document's first three lines (separate_header_and_body), and it is prepended to the first chunk so each chunk stays within max_tokens.

When the header tokenises to >= max_tokens that step becomes non-positive, and Python's negative list slicing then silently corrupts the output:

• an oversized first chunk (header + a huge negative-indexed body slice),
• an empty chunk (token_count == 0),
• and the body gets re-chunked / duplicated from the start.

This is reachable in normal operation because max_tokens and duplicate_headers come straight from user config (cfg.chunking.* in application/worker.py). A small configured chunk size plus a document whose first three lines are long (long title / metadata / front-matter) is enough to trigger it.

While fixing this I also noticed duplicate_headers was a no-op past the first chunk: header_tokens was reset to [] at the end of every iteration, so the self.duplicate_headers or part_index == 0 condition could never include the header again.

Repro

max_tokens=20, a 36-token header, 61 tokens of body:

	before	after
chunks	5	4
token counts	[81, 0, 20, 20, 17]	[37, 20, 20, 20]
empty chunks	1	0
largest chunk	81 (4× the limit)	37 (header + 1 body token — the minimum possible when the header alone exceeds the limit)

duplicate_headers=True, header TITLE\nAUTHOR\nDATE\n:

	before	after
chunks containing the header	1 / 5	6 / 6

The fix

split_document now mirrors the robustness already present in the sibling strategies in chunking_strategies.py (which clamp max_tokens = max(1, …) and never emit empty pieces):

• Reserve header space per chunk with a floor of one body token, so the loop always makes forward progress even when the header alone meets or exceeds max_tokens.
• Include the header on every chunk when duplicate_headers is set, and only on the first chunk otherwise — default behaviour (duplicate_headers=False) is unchanged and produces byte-identical chunks to before for the normal header < max_tokens path.

-            end_position = current_position + self.max_tokens - len(header_tokens)
-            chunk_tokens = (header_tokens + body_tokens[current_position:end_position]
-                            if self.duplicate_headers or part_index == 0 else body_tokens[current_position:end_position])
+            include_header = self.duplicate_headers or part_index == 0
+            active_header = header_tokens if include_header else []
+            body_capacity = max(self.max_tokens - len(active_header), 1)
+            end_position = current_position + body_capacity
+            chunk_tokens = active_header + body_tokens[current_position:end_position]

Tests

Added 4 regression tests in tests/parser/test_chunking.py (TestSplitDocumentHeaderEdgeCases) covering the large-header edge case (token conservation, no empty / corrupted-oversize chunks) and both duplicate_headers modes.

The two key tests fail on main and pass with this change; the full file stays green:

$ python -m pytest tests/parser/test_chunking.py
...
28 passed

The large-header tests fail against the unpatched code, confirming they pin the bug:

FAILED ...::test_header_larger_than_max_tokens_conserves_body   - assert all(tc > 0 ...)  # empty chunk
FAILED ...::test_duplicate_headers_repeats_header_on_every_chunk - assert all("TITLE" in d.text ...)

No UI surface for this change — it's pure backend chunking logic, so the pytest output above stands in for screenshots. ruff check passes on both files.

[vercel]

@FraanW is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

[sviat-barbutsa]

Nice fix. I think one closely related edge case is still worth covering: when the first three newline-terminated lines are the whole document.

There is a small nuance that in that case separate_header_and_body can put all content into header, leaving body_tokens empty. Since split_document only emits chunks inside while current_position < len(body_tokens), a header-only document that is too large for the pass-through path could still return [] and get dropped by classic_chunk.

Could you, please, add a regression test for a long three-line/header-only document, maybe with trailing newline, and make sure the header content is emitted at least once?