fix: extend predict_merges patterns for soft hyphen and Unicode letters

[Frank-Schruefer]

Problem

The merge-continuation heuristic in predict_merges uses two patterns:

• m1: checks that the current element ends with a continuation character [a-z,\-]
• m2: checks that the next element starts with a lowercase letter [a-z]

Two gaps prevent valid merges:

1. Soft hyphen (U+00AD) not handled

Typeset PDFs (books, journals) use U+00AD (SOFT HYPHEN) for optional line-break hyphenation. The character is invisible in rendered text but present in the PDF text layer. A line ending with U+00AD is unambiguously a mid-word line break, but the original pattern [a-z,\-] does not match it — so the two halves of the hyphenated word end up as separate paragraphs in the output.

2. m2 rejects uppercase continuation lines

m2 only accepted [a-z], so continuation lines starting with an uppercase letter were never merged. This affects:

• German nouns (always uppercase)
• Lines after hard line-breaks in typeset text where the next word happens to be capitalised
• Any mid-sentence line-break where m1 clearly indicates no sentence end (no ., !, ?)

Since m1 already ensures the previous line did not end with sentence-ending punctuation, allowing uppercase in m2 is safe.

Fix

# before
m1 = re.fullmatch(r".+([a-z,\-])(\s*)", elem.text)
m2 = re.fullmatch(r"(\s*[a-z])(.+)", sorted_elements[ind_p1].text)

# after
m1 = re.fullmatch(r".+([a-z,\-\u00AD])(\s*)", elem.text)
m2 = re.fullmatch(r"(\s*[a-zA-Z\u00C0-\u024F])(.+)", sorted_elements[ind_p1].text)

[github-actions]

✅ DCO Check Passed

Thanks @Frank-Schruefer, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[nikos-livathinos]

Looks good to me. But the code should be properly formatted and pass all code-checks.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!