Issue concerning formula detection/processing.

[mkd-enrique]

I apologise in advance if this is uploaded in the wrong repository since I was utilising both langchain-opendataloader-pdf and opendataloader-pdf[hybrid]. If it is issued in the wrong repository, kindly inform to me and I will re-issue in the right repository instead!

Bug

When using hybrid_mode="full" on the client side as recommended when the backend is started with --enrich-formula, all pages are expected to be routed to the hybrid backend. However, the extracted output is identical to what is produced without hybrid_mode="full", indicating that pages are still being processed locally and the --enrich-formula enrichment has no effect.

This is especially visible on pages containing mathematical formulas, where the output still contains fragmented SymbolMT-encoded characters instead of properly reconstructed formula content with the appropriate "formula" label. At the end of this issue is a screenshot of the formula for further context.

The following line of command was used to create the hybrid instance:
opendataloader-pdf-hybrid --enrich-formula --port 5002

And the loader had the following parameters:

loader = OpenDataLoaderPDFLoader(
        file_path=[str(sample_pdf)],
        format="json",
        quiet=True,
        split_pages=True,
        use_struct_tree=True, 
        table_method="cluster", 
        include_header_footer=False,
        sanitize=False, 

        # Image Handling
        image_output="external", 
        image_dir="/home/<redacted>/opendata_test/imagestore",
        image_format="png",

        # Hybrid Extractions
        hybrid="docling-fast",
        hybrid_mode="full",
        hybrid_url="http://localhost:5002",
        hybrid_timeout="10000",
        hybrid_fallback=True,
    )

...

Version

Python 3.13.9
langchain-opendataloader-pdf 2.0.0
opendataloader-pdf 2.0.1
langchain-text-splitters 1.1.1

Only the following commands were used concerning package installation:

uv pip install "opendataloader-pdf[hybrid]"
uv pip install -U langchain-opendataloader-pdf
uv pip install -U langchain-text-splitters

...

Java version

openjdk 17.0.16 2025-07-15
OpenJDK Runtime Environment (build 17.0.16+8-Ubuntu-0ubuntu122.04.1)
OpenJDK 64-Bit Server VM (build 17.0.16+8-Ubuntu-0ubuntu122.04.1, mixed mode, sharing)

...

Image

[bundolee]

🤖 AI Issue Fix — status: blocked

Analysis: hybrid_mode=full parameter flow verified correct through langchain-opendataloader-pdf → opendataloader-pdf → Java.
Code paths show no bug. Likely cause: server-side configuration/environment issue.
Blocked: need PDF file and server logs to reproduce.

---

Thanks for the detailed report and for including your exact configuration — that's very helpful.

We've traced the full parameter flow from langchain-opendataloader-pdf through to the Java engine and confirmed that hybrid_mode="full" is correctly passed and handled. Since the output appears identical to non-hybrid mode, the most likely cause is that the hybrid server isn't processing the pages as expected — possibly due to the formula enrichment model not loading, or a connectivity issue between the Java engine and the server.

To investigate further, could you help us with the following?

1. Send the PDF file to open.dataloader@hancom.com so we can reproduce the issue with your exact document.

2. Check the hybrid server startup logs — when you run opendataloader-pdf-hybrid --enrich-formula --port 5002, do you see a log line confirming the formula model was loaded successfully?

3. Run without quiet=True temporarily and share the Java-side logs. Specifically, look for lines like:

   • Hybrid mode=full: skipping triage, all pages to backend (confirms full mode activated)
   • Routing: 0 pages to Java, N pages to Backend (confirms pages sent to backend)
   • Any Backend processing failed or fallback warnings

This will help us pinpoint whether the issue is in the server setup, the formula model, or elsewhere.

📎 If you can share a sample file, please send it to open.dataloader@hancom.com.

[bundolee]

Thank you for the detailed report and the sample document! I've received the PDF and added it to our test fixtures for reproduction.

I'll analyze the hybrid routing path with --enrich-formula and work on a fix.

[mkd-enrique]

Hello! Sorry for the extremely late response. I had done as you instructed and disabled quiet, and I came across the following:
Backend processing failed

However, the logs also mentioned an issue related to timeout. Hence, I increased hybrid_timeout from 10000 to 600000 and it worked well. For reference, this is what was extracted (concerning the image attached in my initial issue):

{
    "type": "formula",
    "id": 560,
    "page number": 3,
    "bounding box": [
        108.35,
        404.508,
        436.01,
        462.161
    ],
    "content": "\\lambda _ { T } = & \\frac { 2 \\cdot \\pi } { \\beta } \\, ; \\quad \\beta = \\omega \\sqrt { \\frac { \\mu _ { 0 } \\cdot \\varepsilon } { 2 } } \\cdot \\sqrt { \\sqrt { 1 + \\left ( \\frac { \\sigma } { \\omega \\cdot \\varepsilon } \\right ) ^ { 2 } } + 1 } \\, ; \\text { with} \\\\ \\omega = & 2 \\cdot \\pi \\cdot f \\, , \\quad \\varepsilon = \\varepsilon _ { 0 } \\cdot \\varepsilon _ { r } \\, , \\quad \\varepsilon _ { 0 } = 8 . 8 5 \\times 1 0 ^ { - 1 2 } \\, , \\quad \\mu _ { 0 } = 4 \\cdot \\pi \\times 1 0 ^ { - 7 } \\, ."
}

So it seems it was an timeout issue all along. Thank you regardless for looking into it for me. I do have another issue that occurs very often, but I'll create a separate Issue for that. Thank you.

[bundolee]

🤖 AI Issue Fix — status: resolved

Resolution: user-confirmed — increasing hybrid_timeout from 10000 to 600000 resolved the formula extraction.
No code change needed. Root cause: default timeout too short for formula enrichment model processing.

---

Glad to hear the formula extraction is working correctly now! As you discovered, the --enrich-formula model requires significantly more processing time per page, especially for documents with complex mathematical content. Increasing hybrid_timeout to 600000 (10 minutes) gives the server enough time to complete.

Thank you for following up with the solution — this will help other users encountering similar timeout issues with formula enrichment. If you run into anything with your next issue, feel free to open it anytime.

[bundolee]

Closing — the root cause was insufficient hybrid timeout for formula enrichment processing. The reporter confirmed that increasing hybrid_timeout resolved the issue. Additionally, PR #337 changes the default timeout to unlimited (0), which prevents this class of issues going forward.