feat(pdf): flag invisible text in metadata

[Success6666]

Summary

• add contains_invisible_text metadata for PDF text objects that include render-mode-3 invisible characters
• keep the invisible-text flag scoped to each split pdfminer text snippet so visible snippets from the same container are not overflagged
• preserve optional metadata semantics during chunk consolidation: absent signals stay absent, explicit False stays False, and any True marks the chunk
• preserve the flag when list items are combined and when elements are chunked
• add regression coverage for pdfminer detection, fast PDF partitioning, snippet-level scoping, and chunk metadata consolidation

Refs #4377

Notes

This is additive: extracted text output is unchanged, and the new metadata gives downstream RAG pipelines a signal they can filter, quarantine, or surface in audit logs.

Tests

• uv run --python 3.12 --group test --extra pdf python -m pytest test_unstructured\partition\pdf_image\test_pdfminer_processing.py::test_text_is_embedded test_unstructured\partition\pdf_image\test_pdfminer_processing.py::test_text_contains_invisible_text test_unstructured\partition\pdf_image\test_pdfminer_processing.py::test_split_pdfminer_text_by_paragraph_keeps_invisible_text_scoped_to_snippet test_unstructured\partition\pdf_image\test_pdf.py::test_partition_pdf_fast_marks_invisible_text test_unstructured\chunking\test_base.py::Describe_Chunker::it_forms_ElementMetadata_constructor_kwargs_by_applying_consolidation_strategies test_unstructured\chunking\test_base.py::Describe_Chunker::it_flags_invisible_text_when_any_element_contains_it test_unstructured\chunking\test_base.py::Describe_Chunker::it_omits_invisible_text_when_no_element_reports_it test_unstructured\chunking\test_base.py::Describe_Chunker::it_keeps_invisible_text_false_when_all_reporting_elements_clear_it test_unstructured\documents\test_elements.py::DescribeElementMetadata::it_can_find_the_consolidation_strategy_for_each_of_its_known_fields -q
• uv run --python 3.12 --group lint ruff check unstructured\chunking\base.py unstructured\partition\pdf.py unstructured\partition\pdf_image\pdfminer_processing.py test_unstructured\chunking\test_base.py test_unstructured\partition\pdf_image\test_pdfminer_processing.py
• uv run --python 3.12 --group lint ruff format --check unstructured\chunking\base.py unstructured\partition\pdf.py unstructured\partition\pdf_image\pdfminer_processing.py test_unstructured\chunking\test_base.py test_unstructured\partition\pdf_image\test_pdfminer_processing.py
• git diff --check

[cubic-dev-ai]

1 issue found across 7 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[Copilot]

Pull request overview

Adds an additive metadata signal to PDF text extraction to flag content-stream text that includes PDF render-mode-3 (invisible) characters, and ensures the signal survives downstream element combination and chunk metadata consolidation.

Changes:

• Detect render-mode-3 characters in PDFMiner text objects and attach contains_invisible_text to element metadata.
• Preserve/propagate contains_invisible_text when combining list items and when consolidating chunk metadata (new ConsolidationStrategy.ANY).
• Add regression tests for invisible-text detection (pdfminer), fast PDF partitioning, and chunk consolidation behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
unstructured/partition/pdf.py	Extracts/snippets PDFMiner text while carrying an invisible-text flag; preserves the flag when combining list items.
unstructured/partition/pdf_image/pdfminer_processing.py	Adds invisible-character detection via shared fidelity stats helper.
unstructured/documents/elements.py	Introduces contains_invisible_text metadata field and assigns consolidation strategy ANY.
unstructured/chunking/base.py	Implements ConsolidationStrategy.ANY in chunk metadata consolidation.
test_unstructured/partition/pdf_image/test_pdfminer_processing.py	Adds unit tests for invisible-text detection and paragraph-splitting scoping behavior.
test_unstructured/partition/pdf_image/test_pdf.py	Adds integration test ensuring FAST strategy marks invisible-text elements.
test_unstructured/chunking/test_base.py	Adds test ensuring chunk metadata is flagged when any element contains invisible text.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 4 files (changes from recent commits).

<sub>Shadow auto-approve: would auto-approve. Adds new contains_invisible_text metadata field for PDF invisible text detection, with tests and consolidation strategy. Changes are additive and backward-compatible.

Re-trigger cubic</sub>