Spurious spaces are getting inserted in PDF with Krutidev text

[amankhandelia]

Bug Description

opendataloader-pdf inserts spurious spaces in Krutidev-encoded Hindi text, breaking character sequences and corrupting the extracted text.

Environment

• opendataloader-pdf version: 2.0.2

Steps to Reproduce

1. Extract page 2 from the PDF using opendataloader-pdf
2. Compare with pypdf extraction

Repro and Logs

$ uv run python -c "
import tempfile
from pathlib import Path
import opendataloader_pdf

with tempfile.TemporaryDirectory() as tmpdir:
    opendataloader_pdf.convert(
        input_path=['page2.pdf'],
        output_dir=tmpdir,
        format='markdown',
        markdown_page_separator='<!-- PAGE %page-number% -->',
        quiet=False,
    )
    
    md_files = list(Path(tmpdir).glob('*.md'))
    if md_files:
        content = md_files[0].read_text(encoding='utf-8')
        print(content)
" 2>/dev/null

Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor preprocessing
INFO: File name: /Users/amank/general/projects/decode_samvidhan/page2.pdf
Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor calculateDocumentInfo
INFO: Number of pages: 1
Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor calculateDocumentInfo
INFO: Author: null
Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor calculateDocumentInfo
INFO: Title: null
Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor calculateDocumentInfo
INFO: Creation date: null
Mar 20, 2026 1:29:22 PM org.opendataloader.pdf.processors.DocumentProcessor calculateDocumentInfo
INFO: Modification date: null
Mar 20, 2026 1:29:23 PM org.opendataloader.pdf.markdown.MarkdownGenerator writeToMarkdown
INFO: Created /var/folders/vn/wd_f5y9s75vdxl4_wd5cs2tw0000gn/T/tmpc5ztcqg2/page2.md
<!-- PAGE 1 -->

# izFke eqnz.k 1950 iqu% eqnz.k 1994 iqu% eqnz.k 2015

# ewY;% ``` 4000@&

## © 2015 yksd lHkk lfpoky;

ykds lHkk d s ifz Ø;k vkjS dk; Z lpa kyu fu;e (iUngz ok a lLa dj.k) d s fu;e 382 d svarxZr izdkf'kr rFkk tSudks vkVZ bafM;k] 13@10] MCY;w-bZ-,-] djksy ckx] ubZ fnYyh&110005 }kjk eqfnzrA

Expected Behavior

pypdf extracts correctly:

yksd lHkk ds izfØ;k vkSj dk;Z lapkyu fu;e (iUnzgoka laLdj.k) ds fu;e 382 ds varxZr izdkf'kr

Actual Behavior

opendataloader-pdf extracts with spurious spaces and character reordering:

ykds lHkk d s ifz Ø;k vkjS dk; Z lpa kyu fu;e (iUngz ok a lLa dj.k) d s fu;e 382 d svarxZr izdkf'kr

Test File

Attached: buggy.pdf

This page contains the problematic Krutidev-encoded Hindi text that demonstrates the bug.

[bundolee]

🤖 AI Issue Fix — status: in-progress

Analysis: single issue, no split needed. Proceeding to fix.

Root cause: verapdf ChunkParser splits text into character-level chunks whose bounding box X positions don't match logical reading order for Krutidev (Walkman-Chanakya) Type1 fonts. TextLineProcessor sorts chunks by X coordinate and inserts spaces based on gap threshold, causing character reordering and spurious spaces.

---

Thanks for reporting this with a clear reproduction case! The Krutidev-encoded Hindi text is being scrambled because our text extraction pipeline sorts individual characters by their visual position, but Krutidev fonts use positioning that doesn't match strict left-to-right order. Working on a fix now.

[bundolee]

🤖 AI Issue Fix — status: resolved

Branch: issue/324-krutidev-spurious-spaces
Changes: Add GlyphReorderProcessor to detect and fix character reordering from verapdf's endX-based sort
Tests: 11 added (10 unit + 1 regression), full suite passing (237 tests)
PR: pending review

---

Thanks for reporting this — you described the problem clearly: Krutidev-encoded Hindi text was coming out garbled with spurious spaces and reordered characters, while pypdf extracted it correctly.

The root cause turned out to be in verapdf's internal character sorting (by glyph endX coordinate rather than stream order), which only misbehaves with fonts that have highly variable glyph widths like Walkman-Chanakya. We've added a new processing step that detects when character positions are out of order and restores the correct sequence using glyph midpoints. A PR with the fix is coming shortly.

[bundolee]

@MaximPlusov Assigning this to you — the root cause is in verapdf's TextPieceComparator, so it's best fixed at the engine level.

Root cause analysis

TextPieceComparator sorts TextPiece objects by endX coordinate (primary key). In ChunkParser.parseString(), each character's position is computed as:

startX = currentX
endX   = currentX + (font.getWidth(code) * fontSize / 1000 * Th)
cursor advance: currentX += (Tc + Tw_if_space) * Th

For this PDF, Tc = -0.0198, fontSize = 12. Because |Tc| is tiny, all startX values cluster within ~0.2pt of each other. Meanwhile endX varies by several points depending on glyph width. The endX-based sort effectively reorders characters by glyph width instead of stream order.

This only affects fonts with highly variable glyph widths (e.g., Walkman-Chanakya / Krutidev Hindi fonts). For most fonts, glyph widths are uniform enough that endX ordering coincides with stream ordering.

Example

Expected (pypdf / correct stream order):

yksd lHkk ds izfØ;k vkSj dk;Z lapkyu

Actual (endX-sorted):

ykds lHkk d s ifz Ø;k vkjS dk; Z lpa kyu

Suggested fix

In TextPieceComparator, change the primary sort key from endX to startX (or preserve PDF stream order). This would fix the issue for all affected fonts without downstream workarounds.

Reproduction

Test PDF and regression test are on branch issue/324-krutidev-spurious-spaces:

• Test fixture: java/opendataloader-pdf-core/src/test/resources/regression/issue324-krutidev.pdf
• Regression test: KrutidevSpacingDiagnosticTest — asserts output contains "yksd lHkk ds"

[MaximPlusov]

@bundolee We fixed the TextPieceComparator in the latest verapdf version.

[amankhandelia]

Hey @MaximPlusov I do see a commit for the fix, but is there a PR I can refer to

Just curious to know how these things work.

[MaximPlusov]

@amankhandelia There is no PR for this fix.

[bundolee]

This is now fixed. The root cause was in veraPDF's TextPieceComparator (sorting by endX instead of startX), which @MaximPlusov fixed in veraPDF/veraPDF-validation@dc95769.

Verification — extracted your test PDF (page2.pdf) with the current build (verapdf wcag-validation 1.31.33):

Text	Before (reported)	After (current)
Body paragraph	ykds lHkk d s ifz Ø;k vkjS dk; Z lpa kyu	yksd lHkk ds izfØ;k vkSj dk;Z lapkyu ✅

Output now matches the expected pypdf extraction — no more spurious spaces or character reordering.

The fix will be included in the next opendataloader-pdf release.

---

🤖 AI Issue Fix — status: resolved

Root cause: veraPDF TextPieceComparator sorted TextPiece by endX, reordering characters with variable glyph widths (Krutidev/Walkman-Chanakya fonts). Fixed upstream in veraPDF v1.31.29+, now resolved in opendataloader-pdf via dependency range [1.31.0, 1.32.0-RC) → 1.31.33.