feat: add max_page to chunk_by_title and remove multipage_sections

[issahammoud]

Summary

Adds a max_page parameter to chunk_by_title() that gives callers a hard upper bound on how many pages a single chunk may span. max_page is a strict superset of the existing multipage_sections flag: max_page=None (default) matches multipage_sections=True, and max_page=1 matches multipage_sections=False — plus the new ability to express any page-count limit (max_page=2, max_page=3, …) that the old binary flag could not.

multipage_sections is kept for backward compatibility but deprecated. It emits a DeprecationWarning and will be removed in a future release.

Motivation

chunk_by_title supported character/token-based size limits and an all-or-nothing page boundary toggle (multipage_sections). There was no way to say "allow cross-page sections, but cap each chunk at N pages". This is a common need in RAG pipelines where page-level context windows matter — e.g. a retriever that attaches page images to each chunk should not pull more than a fixed number of page images per chunk.

Behaviour

When max_page=N is specified, the chunker tracks the page where each chunk began and starts a new chunk as soon as an element's page number is N or more pages past that start page:

chunks = chunk_by_title(elements, max_page=2)
# → each chunk spans at most 2 pages

chunks = chunk_by_title(elements, max_page=1)
# → a new chunk starts on every page change

• Additive constraint — works alongside all existing by_title boundaries (title, size, token).
• Title-aware reset — a Title element resets the page counter without double-firing, so each title-delimited section's page span is measured independently.
• Hard limit, not a hint — PreChunk.can_combine() also enforces max_page, so PreChunkCombiner cannot silently re-merge pre-chunks that were split by the page boundary.
• No page-number, no problem — elements without metadata.page_number are assumed to continue the current page.
• Validation — max_page < 1 raises ValueError.

multipage_sections deprecation

multipage_sections is kept but deprecated. Existing call sites continue to work without modification; they will receive a DeprecationWarning pointing to the max_page equivalent.

Old	New	Notes
multipage_sections=True (default)	omit max_page (default)	No code change needed
multipage_sections=False	max_page=1	One-word change

When both multipage_sections and max_page are provided, max_page takes priority silently — no error is raised.

Example

from unstructured.chunking.title import chunk_by_title
from unstructured.documents.elements import Title, Text, ElementMetadata

elements = [
    Title("Introduction",  metadata=ElementMetadata(page_number=1)),
    Text("Page 1 content.", metadata=ElementMetadata(page_number=1)),
    Text("Page 2 content.", metadata=ElementMetadata(page_number=2)),
    Text("Page 3 content.", metadata=ElementMetadata(page_number=3)),
    Text("Page 4 content.", metadata=ElementMetadata(page_number=4)),
    Text("Page 5 content.", metadata=ElementMetadata(page_number=5)),
]

chunks = chunk_by_title(elements, max_page=2, combine_text_under_n_chars=0)
# chunk 0 → pages 1-2  (Introduction + Page 1 + Page 2)
# chunk 1 → pages 3-4  (Page 3 + Page 4)
# chunk 2 → page  5    (Page 5)

Bug fix: PreChunkCombiner was undoing max_page boundaries

Root cause: PreChunker correctly split elements using is_on_page_exceeding_max, but PreChunkCombiner only checked combine_text_under_n_chars and hard_max when deciding whether to merge consecutive pre-chunks. Two pre-chunks split by a max_page boundary could be silently re-merged if their combined text was small enough, making max_page a soft hint rather than a hard limit.

Fix:

• Added a max_page hook to ChunkingOptions base (returns None; overridden by _ByTitleChunkingOptions) so PreChunk.can_combine() can enforce the limit without coupling to a specific strategy.
• Added PreChunk._page_span — a cached property returning (first_page, last_page) for elements that carry a page number.
• Guarded PreChunk.can_combine() to block combination whenever the merged span would exceed max_page pages.

Changes

unstructured/chunking/base.py

• New is_on_page_exceeding_max(max_page) boundary predicate factory — handles all page-boundary cases including max_page=1.
• ChunkingOptions.max_page — new cached property returning None by default; overridden by _ByTitleChunkingOptions.
• PreChunk._page_span — cached property returning (first_page, last_page).
• PreChunk.can_combine() — page-span guard blocks combination that would violate max_page.

unstructured/chunking/title.py

• chunk_by_title() gains max_page: Optional[int] = None. multipage_sections is kept but emits DeprecationWarning when passed.
• _ByTitleChunkingOptions.max_page — reads max_page kwarg directly; falls back to translating multipage_sections=False → 1 when max_page is not set. max_page takes priority when both are provided.
• _ByTitleChunkingOptions.multipage_sections — kept as a read-only backward-compat alias (max_page != 1).
• _ByTitleChunkingOptions.boundary_predicates — uses is_title plus is_on_page_exceeding_max when max_page is set; is_on_next_page() removed.
• _ByTitleChunkingOptions._validate() — rejects max_page < 1.

unstructured/chunking/dispatch.py

• Help text updated to advertise max_page (by_title only) and mark multipage_sections as deprecated.

Structural diff

%%{init: {'layout': 'elk', 'elk': {'direction': 'RIGHT'}, 'maxTextSize': 999999, 'theme': 'base', 'themeVariables': {'background': '#ffffff', 'clusterBkg': '#f8fafc', 'clusterBorder': '#94a3b8', 'primaryColor': '#f8fafc', 'primaryBorderColor': '#94a3b8', 'primaryTextColor': '#1e293b', 'lineColor': '#64748b', 'fontSize': '13px', 'fontFamily': 'ui-monospace, SFMono-Regular, Menlo, monospace'}}}%%
classDiagram
    direction LR

    class n2["_ByTitleChunkingOptions"] {
        <<unstructured/chunking/title.py>>
        + max_page()
        ~ _validate()
        ~ boundary_predicates()
        ~ iter_boundary_predicates()
        ~ multipage_sections()
    }
    class n5["unstructured/chunking/title.py"] {
        ~ chunk_by_title()  sig
    }

    class n0["ChunkingOptions"] {
        <<unstructured/chunking/base.py>>
        + max_page()
    }
    class n1["PreChunk"] {
        <<unstructured/chunking/base.py>>
        + _page_span()
        ~ can_combine()  calls +1
    }
    class n3["unstructured/chunking/base.py"] {
        + is_on_page_exceeding_max()
        + page_count_exceeded()
    }

    style n2 fill:#fefce8,color:#854d0e,stroke:#f59e0b,stroke-width:3px
    style n5 fill:#fefce8,color:#854d0e,stroke:#f59e0b,stroke-width:3px
    style n0 fill:#f0fdf4,color:#166534,stroke:#22c55e,stroke-width:3px
    style n1 fill:#fefce8,color:#854d0e,stroke:#f59e0b,stroke-width:3px
    style n3 fill:#f0fdf4,color:#166534,stroke:#22c55e,stroke-width:3px

    n2 --|> n0
    n2 --> n3 : calls

Loading

• Green — new (ChunkingOptions.max_page, is_on_page_exceeding_max, page_count_exceeded)
• Yellow — modified (_ByTitleChunkingOptions, PreChunk, chunk_by_title signature, dispatch.py)
• _ByTitleChunkingOptions --|> ChunkingOptions — inherits and overrides max_page
• _ByTitleChunkingOptions --> base.py — calls is_on_page_exceeding_max; is_on_next_page removed
• multipage_sections() remains on _ByTitleChunkingOptions as a read-only backward-compat alias

Testing

pytest test_unstructured/chunking/test_title.py -q
# 60 passed, 7 skipped

New / updated tests:

• test_max_page_1_is_equivalent_to_multipage_sections_false — proves max_page=1 and the deprecated multipage_sections=False produce identical output
• test_chunk_by_title_respects_max_page — chunks split at correct page boundaries
• test_chunk_by_title_max_page_resets_on_title — Title resets the counter independently per section
• test_chunk_by_title_max_page_1_breaks_on_every_page — max_page=1 breaks on every page change
• test_chunk_by_title_max_page_not_undone_by_combiner — regression: combiner cannot re-merge max_page-split pre-chunks
• test_chunk_by_title_max_page_none_does_not_add_page_boundary — omitting max_page leaves existing behaviour unchanged
• it_translates_multipage_sections_false_to_max_page_1 — internal translation verified
• it_leaves_max_page_none_when_multipage_sections_is_true — no-op case verified
• it_prefers_max_page_over_multipage_sections_when_both_are_set — priority rule verified
• it_emits_deprecation_warning_when_multipage_sections_is_used — warning verified
• it_does_not_warn_when_multipage_sections_is_not_passed — no spurious warnings
• it_accepts_valid_max_page_values — parametrized: 1, 3, None
• it_rejects_max_page_less_than_one — parametrized: 0, -1, -10
• Existing multipage_sections integration tests wrapped with pytest.warns(DeprecationWarning)

Checklist

• New functions have docstrings and type hints
• Tests added and passing (60 passed, 7 skipped)
• CHANGELOG.md updated
• __version__.py bumped to 0.23.4
• No debugging artifacts or commented-out code

[cubic-dev-ai]

1 issue found across 5 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[issahammoud]

Thanks for catching this — the issue is valid.

Root cause: PreChunkCombiner only checked combine_text_under_n_chars and hard_max when deciding whether to merge consecutive pre-chunks. Pre-chunks split by is_on_page_exceeding_max could be silently re-merged if their combined text was small enough, making max_page a soft hint rather than a hard limit.

Fix (commit 39b24c2):

• Added a max_page hook to ChunkingOptions base (returns None; overridden by _ByTitleChunkingOptions). This lets PreChunk.can_combine() consult the limit without coupling to a specific strategy.
• Added PreChunk._page_span — a cached property returning (first_page, last_page) for elements that carry a page number.
• Guarded PreChunk.can_combine() with a page-span check: blocks combination when the merged span would exceed max_page pages.
• Added a regression test (test_chunk_by_title_max_page_not_undone_by_combiner) that exercises this path without suppressing the combiner via combine_text_under_n_chars=0, confirming the guard is what holds the limit.

[cubic-dev-ai]

0 issues found across 3 files (changes from recent commits).

<sub>Shadow auto-approve: would require human review. Adds a new max_page parameter to chunking logic, modifying core PreChunk combination and boundary detection. While well-tested, this feature introduces new stateful predicates and changes to core chunking paths, which carries moderate risk.

Re-trigger cubic</sub>

[cubic-dev-ai]

2 issues found across 3 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would require human review. This PR changes core chunking logic that could affect RAG output quality and correctness. Because of the complexity and risk of regression, it warrants human review.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured/chunking/title.py">

<violation number="1" location="unstructured/chunking/title.py:31">
P1: Custom agent: **Flag AI Slop and Fabricated Changes**

PR description claims multipage_sections is 'removed entirely' and 'passing it now raises TypeError', but the code adds it back as an optional parameter with a DeprecationWarning and backward-compatibility logic instead.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Custom agent: Flag AI Slop and Fabricated Changes

PR description claims multipage_sections is 'removed entirely' and 'passing it now raises TypeError', but the code adds it back as an optional parameter with a DeprecationWarning and backward-compatibility logic instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/chunking/title.py, line 31:

<comment>PR description claims multipage_sections is 'removed entirely' and 'passing it now raises TypeError', but the code adds it back as an optional parameter with a DeprecationWarning and backward-compatibility logic instead.</comment>

<file context>
@@ -27,6 +28,7 @@ def chunk_by_title(
     max_characters: Optional[int] = None,
     max_page: Optional[int] = None,
     max_tokens: Optional[int] = None,
+    multipage_sections: Optional[bool] = None,
     new_after_n_chars: Optional[int] = None,
     new_after_n_tokens: Optional[int] = None,
</file context>

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic</sub>