feat: Add docling-serve integration#3139
Draft
cbrumm wants to merge 7 commits intodeepset-ai:mainfrom
Draft
Conversation
|
|
github-actions
bot
added
topic:CI
type:documentation
Add a new DoclingServeConverter component that converts documents via a running docling-serve REST API instance, avoiding docling's heavy local dependencies (PyTorch, etc.). Relates to deepset-ai#2960 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cbrumm
force-pushed
the
feature/docling-serve-integration
branch
from
ccb9443 to
ec97ce5
Compare
The API reference build expects docling_serve.md (matching the integration folder name), not docling-serve.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pytest exits with code 5 when no tests are collected. The integration test step selects only @pytest.mark.integration tests, which were missing. Add a skipped integration test that runs when DOCLING_SERVE_URL is set. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… conversion
The /v1/convert/source endpoint requires the new discriminated union format
with "sources": [{"kind": "http", "url": ...}] instead of the deprecated
"http_sources" field. Updated both sync and async code paths and tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Start docling-serve-cpu container on Linux runners and run integration tests against it. Adds a URL conversion test to verify the v1 sources format fix end-to-end. macOS/Windows runners skip integration tests since Docker services are not available. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ptions Create a single httpx.Client/AsyncClient per run() call instead of one per source, avoiding unnecessary connection overhead when converting multiple documents. Also copy the convert_options dict on init to prevent external mutation from affecting component behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft created at PyCon DE Sprints — feedback and review welcome!
Relates to #2960
DoclingServeConvertercomponent that converts documents via a running docling-serve REST API instancedocling-haystackintegration to avoid heavy local dependencies (PyTorch, etc.)ByteStreamobjects, and URLs as sourcesrun()(sync viahttpx.Client) andrun_async()(viahttpx.AsyncClient)docling-serve-cpuDocker container on Linux CI runnersNote that
docling-serverdocker images are quite large, I think around 4 GB! See pypi docling-serve.Design decisions
base_url,api_key(Secret),timeout,convert_options(single dict for all docling-serve parameters)/v1/convert/source(v1sourcesformat withkinddiscriminator), local files and ByteStreams to/v1/convert/filehttpxfor both sync and async HTTP in a single dependencyghcr.io/docling-project/docling-serve-cpuviadocker runon Linux runners, with health-check polling. macOS/Windows skip integration tests.Test plan
hatch run fmt— all checks passedhatch run test:types— no issues foundhatch run test:unit— 30/30 passed/v1/convert/file/v1/convert/source(validates v1sourcesformat)🤖 Generated with Claude Code